- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ProtoAda proposes format-aware task grouping, selective LoRA expansion, and geometric consolidation for rehearsal-free multimodal continual instruction tuning. Experiments on TriGap and UCIT, including controlled format shifts and a grounding extension, evaluate its accuracy, efficiency, and ability to preserve response protocols.
Critical strengths
-
The format-aware task grouping is well motivated. Incorporating response-format statistics into task prototypes provides a coherent way to account for output compatibility when sharing adaptation parameters. The improvement from in Table 3 together with the advantage over late fusion in Table 5, supports the usefulness of this design.
-
The method demonstrate a useful balance between accuracy and adaptation cost. Compared with SAME + format, it uses 16.4% fewer stored parameters and 22.3% less training time, with accuracy point estimates of 74.66 versus 74.92 in Table 7.
Critical weaknesses
- The consolidation mechanism needs a precise operational definition.
It is unclear whether masking, projection and SVD act on LoRA-factor increments or effective weight updates, and how group bases are initialized and updated in Eq. (3), (14)–(21). This leaves a central component difficult to reproduce and assess.
- Reporting inconsistencies need clarification.
The mean task score in Table 1 is 46.928, whereas the reported value is 47.23. In Table 6, the heading referring to three tasks conflicts with the average value calculated for six tasks.
Questions for authors
-
Which exact tensors and shapes implement Eqs. (14)–(21), how is initialized and updated
-
Which Table 1 entries or aggregate are incorrect? Does Table 6 accuracy cover three or six tasks?
-
Which dataset and scoring rule were actually used for FmtGap?
6 comments
Q4. Which dataset and scoring rule were used for FmtGap?
The intended dataset is Flickr30k throughout FmtGap. The five phases change the required response protocol: brief description, detailed description, short/one-word answer, multiple-choice answer, and yes/no answer. The references to VizWiz in the FmtGap construction descriptions will be corrected; VizWiz remains part of the semantic-varied VisGap comparison.
The Task-0 drop is defined as
using the same held-out Task-0 samples and reference captions before and after subsequent training. Task 0 uses the standard UCIT caption-evaluation pipeline, implemented in backbone/shared/eval/eval_caption.py with COCOEvalCap. Generated captions and reference captions are PTB-tokenized, and the evaluator computes BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, ROUGE-L, and CIDEr. Each score is multiplied by 100, and is the unweighted arithmetic mean of these seven scaled scores:
where the terms inside the parentheses are the raw evaluator outputs. For illustration, a 3,000-sample caption-evaluation log reports the following scaled scores:
| BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | CIDEr | Mean |
|---|---|---|---|---|---|---|---|
| 72.79 | 57.68 | 46.08 | 37.73 | 27.47 | 55.66 | 143.04 | 62.92 |
This example illustrates the scoring rule, not an additional FmtGap performance result. Although the script prints the mean with a percent sign (Average: 62.92%), it is a composite caption-quality score, not the percentage of correctly answered samples. No threshold is applied to convert caption similarity into binary correctness, and scaled CIDEr can exceed 100. Accordingly, is reported in caption-score points, not percentage points of sample-level accuracy.
Task-0 retention after each subsequent phase is evaluated under the original Task-0 caption protocol and with this same metric; the later phases' response formats do not change the Task-0 scoring rule. We will specify this aggregation in the FmtGap evaluation description and distinguish caption quality from content correctness and format compliance.
Dear Reviewer jbGD,
Thank you again for your thoughtful and constructive feedback. We have tried to address your concerns regarding the shape of tensors, clarifications of actual accuracy, Table 6 and FmtGap.
As the discussion period is approaching its end, we would greatly appreciate any final feedback if time permits. If there are any remaining concerns or points that you would like us to clarify further, we would be happy to respond.
Best regards,
Authors
Response to Reviewer jbGD
Thank you for recognizing the motivation for protocol-aware grouping and the accuracy–cost trade-off. We address each requested clarification below.
Q1. Exact tensors, shapes, and subspace initialization in Eqs. (14)–(21)
These operations are performed on effective weight matrices, not directly on LoRA-factor increments. For each linear module:
For nonzero Gg, the current group subspace is the one-dimensional span of vec(Gg), and the shared component is ⟨D,Gg⟩FGg/∥Gg∥F2. There is no separately maintained multidimensional group basis. An empty group is initialized from the full masked task update, with zero initial residual. Subsequent tasks update the effective group matrix through weighted absorption and low-rank storage; the next projection uses that updated matrix.
Global promotion adds a common matrix H to global memory and subtracts it from each group before compression. Residual pruning, in contrast, is approximate deletion based on a coverage criterion and does not retain a separate task-specific coefficient vector. We will make these distinctions, the scaling conventions, and the numerical approximations explicit in the revised equations and pseudocode.
Q2. Which Table 1 entries or aggregate are incorrect?
The errors are in two per-task entries: FloodNetVQA should be 80.96, not 74.03, and CLEVR-Math should be 54.00, not 57.87. With the remaining eight scores unchanged, the corrected sum is 472.34 and the mean is 47.234%, which rounds to the reported 47.23%. Thus, the aggregate is consistent with the corrected task scores; the two task entries need to be corrected. The corresponding improvement over DISCO's listed-score average of 46.544% is 0.690 percentage points. We apologize for the transcription errors.
Q3. Does Table 6 cover three tasks or six?
Protocol compliance is averaged over three tasks: ArxivQA, IconQA, and CLEVR, whose required answer forms admit reliable deterministic checks; the other tasks do not admit the same unambiguous protocol check. The accuracy column is the six-task UCIT final average, copied from Table 2 as an overall performance reference, and remains unchanged. We will explicitly label these different task scopes and explain why compliance is restricted to three tasks in the caption. The original presentation conflates these scopes, and we apologize for the ambiguity.