- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies compositional continual visual question answering. It keeps a frozen, task-specific pair of low-rank VQA and visual-question-generation experts for each sequential task. When learning a new task, earlier experts generate filtered question–answer turns on the current task's images, and the new expert learns from these additional views. At inference, a language-only router selects one expert. The paper also introduces a 500-example human-annotated compositional test set and a 41,981-question COCO-GQA split. On LLaVA-7B, final average performance on VQACL is 61.20%, versus 54.50% for the strongest listed continual baseline; accuracy on the human compositional set is 51.40%, versus 43.20% for GaB.
Critical strengths
S1. The paper identifies a worthwhile gap between retaining individual VQA skills and answering questions that require several learned skills. The additional human-annotated test set records intended task combinations, which gives the continual-learning setting a more direct compositional evaluation than per-task accuracy alone (Section 5). S2. The reported gains are substantial across the main evaluations. The LLaVA-7B model reaches 61.20% final average performance on VQACL, 64.40% on COCO-GQA, and 51.40% on the human compositional set (Table 1 and Figure 3); the BLIP-2 appendix also reports the highest average performance among its listed baselines (Table 8). The Socratic ablation improves COCO-GQA by 9.87 points and the human set by 6.00 points (Table 5 and Appendix Table 11), making the proposed supervision worth further study. S3. The method gives a concrete way to reuse prior experts without retaining raw training images. Freezing earlier adapters protects their task-specific parameters, while generated turns provide training-time cross-task views. The paper also tests a final-expert-only variant and a direct two-adapter merge (Appendix Tables 9–10), which helps clarify why it uses a routed expert pool at inference.
Critical weaknesses
W1. The task-agnostic routing evidence does not yet cover the paper's central compositional setting. The router is trained to assign one task label to each question, and Figure 3(c) evaluates that prediction on single-task VQACL questions. A compositional question has two or more intended task labels, but inference still selects only one expert. Although Appendix Table 10 shows that the full pool beats the final expert overall, it does not show which expert receives each composition type or whether that expert was trained after all required skills became available. Errors from routing and errors from the selected expert are therefore difficult to separate. Please analyze routing on multi-skill questions and performance by selected expert and task pair. W2. The ablations do not fully isolate Socratic questioning from more training. Table 5 removes generated turns entirely. Table 4 matches the number of question–answer pairs, but changes current versus past images and studies only the last task with a single expert. Its gains are modest (51.45 versus 49.61 on VQACL; 61.76 versus 60.80 on COCO-GQA), and the human set is not evaluated. A matched current-image control with equal turns and tokens but generic, shuffled, or single-expert questions would test whether complementary prior-task views cause the compositional gain. Earlier VQA experts' pseudo-answer accuracy also needs measurement. W3. The new benchmark is promising, but its validity for skill composition needs stronger evidence. It has 500 questions on 418 images across 100 composition types (Section 5), with uneven type frequencies in Figure 2. Annotators saw existing VQACL question–answer triples; low-confidence items were filtered using self-ratings and peer review of a random subset. The paper does not report independent verification that each accepted question truly requires every assigned skill, agreement on task labels, or answer accuracy after annotation. COCO-GQA adds scale, but its scene-graph questions are not labeled as compositions of VQACL task types (Appendix F–G). Per-type results, an independent audit, and uncertainty intervals would make the compositional conclusions more secure. W4. The storage comparison in Table 7 is not made at the same point in the task sequence. It contrasts a 770 MB replay buffer with about 233 MB of expert parameters per task; at ten tasks, the stated expert cost alone is about 2.33 GB, before the router and shared backbone. Thus, the table does not show a lower total storage footprint for the reported ten-task experiment. Moreover, each new task queries all earlier VQG and VQA experts on current images, while the paper gives no end-to-end training time, inference time, or peak-memory comparison. These totals are needed to assess the claimed practical overhead of the expandable design.
Questions for authors
Q1. For W1, how are multi-skill questions routed among single-label experts? Please report expert-selection distributions and accuracy by composition type and learning step, including cases where the selected expert was trained before one required skill was introduced. Q2. For W2, can the authors add equal-turn and equal-token current-image controls using generic or shuffled previous-task questions, and measure the factual correctness of generated pseudo-answers? Reporting these on both compositional benchmarks would identify what Socratic supervision contributes beyond extra training data. Q3. For W3–W4, can the authors independently audit a stratified sample for answer correctness and genuine multi-skill necessity, give per-type or paired-bootstrap uncertainty, and compare total adapter storage and end-to-end training/inference cost at ten tasks with the replay baselines?
We thank the reviewer for taking the time to review our paper.
Q1. Routing of multi-skill questions.
We thank the reviewer for this suggestion. Our framework enables compositional reasoning through Socratic knowledge transfer during training, rather than combining multiple experts at inference. Each expert retains its primary/dominant task specialization while acquiring knowledge from previous tasks. Therefore, our router selects an expert based on the dominant component of a compositional question.
We analyze whether the router selects an expert corresponding to at least one required component task across learning steps.
The router achieves 83.84% average component-task alignment across learning steps. We further compare router-based selection against always selecting the latest required-task expert on the compositional questions.
The router improves compositional accuracy by 3.6 percentage points. This suggests that expert specialization remains relevant despite cross-task knowledge transfer, and selecting dominant / most important skilled expert improves performance. Socratic supervision enables knowledge sharing across experts without requiring identical compositional performance from every expert. Additionally, Tab. 11 provides evidence for Socratic supervision with improved performance on all three benchmarks (average improvement of +6.18 pp).
Q2. Controls beyond extra training data.
Our Replay vs. Socratic questioning (Tab. 4) uses additional replay images for the Replay method, while our method reuses the current task image (i.e matched QAs but additional images for the replay method). We thank the reviewer for the suggested comparison with matched training samples. Following the suggestion, we ran a matched current-image control on the last task. We kept all original last training data and added shuffled questions. So both models see identical questions, answers, image counts, and training settings. Socratic questioning achieved best performance on all ten tasks and both compositional benchmarks. We note that the suggested shuffling pairs QAs with the wrong images, so part of the drop may reflect label noise.
Pseudo-answer correctness.
We checked 200 randomly sampled generated Q&A pairs. An author judged 81.5% of pseudo-answers correct, 15% wrong, and 3.5% ambiguous(unable to answer the question for the corresponding image). We additionally judged the same with pairs using GPT-5.5 as a judge. GPT-5.5 judged 85.5% correct. Pseudo-labels are therefore mostly correct, with little generation noise.
Q3. Audit, uncertainty, and cost.
Audit. An annotator with an independent reviewer audited a stratified sample of 100 questions; reference-answer agreement was 86.0%, and task-label (at least two) agreement was 77.0%.
Uncertainty. A paired bootstrap over questions (10,000 resamples) gives our gain over GaB on COCO-GQA as +5.42 pp (64.40 vs. 58.98; 95% CI [4.97, 5.86]; positive in every resample).
Storage. Our expert pool grows linearly at about 233 MB per task, or 2.33 GB at ten tasks, which exceeds a 770 MB replay buffer. The advantage of our design is that we store no raw past-task images.
Inference cost.
On 100 test questions (10 per task, ~2.5 answer tokens on average), end-to-end inference latency matches a single expert:
Training cost.
End-to-end training time over all ten tasks on three A100 40GB GPUs, including Socratic generation by past experts for our method:
Socratic generation makes training about 1.9× slower than replay, though still faster than CL-MoE. The overhead stays manageable because experts are independent and share the backbone and current-task images. This is a trade-off: the extra cost is paid once during training, while inference—incurred on every query—matches single-model latency and is about 3× faster than CL-MoE. Additionally, our method improved performance (an average improvement of +6.77 pp compared to the second-best baseline) on all benchmarks (VQACL, COCO-GQA, Compositional VQA).
We thank the reviewer for the review, which has strengthened our paper. We will incorporate these analyses and address any remaining minor comments in the revised version.