- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents CATSF, a capability-aware teacher-student framework for adaptive reasoning in multimodal user feedback classification, along with a new Android User Feedback (AUF) dataset containing 3,750 expert-annotated text-image examples across 21 categories. CATSF employs a two-stage fine-tuning strategy: first, a compact student model learns basic and complex reasoning trajectories generated by teacher MLLMs; second, it learns to adaptively select reasoning paths using supervision derived from its own classification correctness under basic reasoning. Experiments with two teacher models and two student models demonstrate improvements over the evaluated classification baselines.
Critical strengths
1.The paper addresses the practical challenge of balancing classification performance and computational efficiency in multimodal user feedback analysis. Considering both input complexity and student-model capability when selecting reasoning depth provides a reasonable motivation for the proposed approach. 2.CATSF introduces a concise two-stage fine-tuning framework that separates inference learning from adaptive inference path selection.
Critical weaknesses
1.Insufficient Motivation and Justification for the New AUF Dataset The motivation and necessity of introducing AUF are not sufficiently justified in the main paper. Existing Android user feedback and bug-report datasets, such as Android Apps and User Feedback [1] and the Android GUI-based bug-report dataset [2], are not adequately discussed or compared. Although these datasets differ from AUF in their tasks and modalities, the paper does not clearly establish what unique challenges AUF addresses or why a new benchmark is necessary. A more comprehensive discussion of existing resources and a clearer justification of AUF's distinctive contributions should be included in the main paper. [1]Grano, Giovanni, et al. "Android apps and user feedback: a dataset for software evolution and quality improvement." Proceedings of the 2nd ACM SIGSOFT international workshop on app market analytics. 2017. [2]Mahmud, Junayed, et al. "On using gui interaction data to improve text retrieval-based bug localization." Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 2024.
2.Limited Comparisons with Relevant Adaptive Reasoning Methods Although Table 2 compares CATSF with several multimodal classification baselines, including BLIP-2, Multimodal-CoT, and WisdoM these methods do not directly address adaptive reasoning-path selection. More closely related approaches, such as Thinkless [3], TON [4], and TATA [5], are discussed in Section 2.2 but are not included in the experimental comparisons. Comparisons with these methods or appropriately adapted variants under comparable backbone and training settings would better establish the effectiveness of the proposed approach. [3] Fang, Gongfan, Xinyin Ma, and Xinchao Wang. "Thinkless: Llm learns when to think." Advances in neural information processing systems 38 (2026): 151268-151295. [4] Wang, Jiaqi, et al. "Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models." Advances in Neural Information Processing Systems 38 (2026): 111196-111234. [5] Xu, Xin, et al. "Teaching LLMs According to Their Aptitude: Adaptive Switching Between CoT and TIR for Mathematical Problem Solving." The Third Conference on Parsimony and Learning (Proceedings Track). 2026.
Questions for authors
- What specific limitations of existing Android user feedback datasets motivate the construction of AUF? What distinctive challenges does AUF introduce compared with existing datasets?
- The anonymous repositories do not appear to provide the image files or retrieval URLs for AUF.
- Tables 3 and 6 demonstrate performance improvements after two-stage fine-tuning. How can the authors distinguish gains from adaptive reasoning-path selection from those resulting from additional supervised training?
- Why are closely related adaptive reasoning methods, such as Thinkless, TON, and TATA, not included in the experimental comparisons?
- Given that capability-aware reasoning-path selection is not specific to Android user feedback, can the authors evaluate CATSF on established multimodal benchmarks to demonstrate its generalizability?
- Why does adaptive reasoning achieve the highest Macro-F1 in only two of the four configurations in Table 3? Can the authors provide per-class performance results and text-only/image-only ablations, together with analyses across the four defined text-image relationship types?
2 comments
Thank you for the additional experiments and clarifications. However, comparisons with closely related adaptive reasoning methods, such as Thinkless, TON, and TATA, remain necessary to demonstrate the effectiveness and competitiveness of CATSF. I will keep my score.
Q1 and Q2 We thank the reviewer for raising this concern. Existing Android feedback datasets lack support for multimodal classification: Android Apps and User Feedback [1] contains primarily textual feedback, while the GUI-based bug-report dataset [2] focuses on bug localization.
AUF addresses these gaps with 3,750 expert-labeled text–image pairs. It also introduces three challenges: unreliable user-provided labels, fine-grained classification across 21 technical categories requiring domain expertise, and heterogeneous text–image relationships requiring cross-modal reasoning.
Q3. We have approximately 1.6 GB of image files, hosted in the Releases section of the original GitHub repository. We have also added example data entries to examples/posts.json.
Q4. We conducted two controlled experiments to disentangle the effects of additional training and capability-aware selection. First, continuing Stage 1 SFT with the same number of updates achieved 81.50%, below CATSF (86.00%), suggesting that additional training alone does not explain the gains. Second, replacing capability-aware selection with random selection under the same Stage 2 training budget achieved 83.50%, compared with 86.00% for CATSF.
Q5. Our evaluation focuses on student-correctness-guided SFT for multimodal feedback classification. As discussed in Sec. 2.2, Thinkless uses RL for reasoning-length selection, TON combines thought-dropout SFT with GRPO, and TATA selects CoT/TIR demonstrations via anchor-set evaluation. In contrast, CATSF selects teacher targets based on the task-adapted student's basic-path correctness and learns selection through a second SFT stage, without RL or tool execution. Fair comparisons would require substantial pipeline adaptations and budget controls, potentially confounding method effects with implementation choices.
Q6. Although evaluated on Android user feedback classification, CATSF's capability-aware reasoning strategy is not inherently domain-specific. Applying it to other multimodal benchmarks would require adapting the reasoning paths and prompts. We leave such evaluations for future work.
Q7. The main reason is the class imbalance in reliable selection supervision. Some categories in the website are extremely underrepresented, such as GPS and Location. Our additional evaluation shows that Gemini-2.5-flash achieves only 0%, 0%, and 66.7% accuracy on these categories, respectively, compared with 50%, 50%, and 100% for GPT-4o-mini. Since CATSF retains only teacher trajectories with correct final predictions, fewer reliable examples from these minority classes are available for selection training under Gemini supervision. Consequently, the learned selection policy is more influenced by majority classes, potentially leading to less reliable reasoning-path choices for minority classes. This favors overall Accuracy but limits Macro-F1.
Q8. We conducted three additional experiments.
First, we evaluated classification accuracy across individual categories. The results show varying performance across categories, with relatively lower accuracy on Accessibility, Location, and Performance. These categories remain more challenging for the model.
Per-class accuracy results are presented below.
Second, we conducted text-only and image-only ablations using Qwen2.5-VL-7B, comparing them with the full CATSF model. The full model achieves 86.0% accuracy, outperforming the text-only and image-only variants by 6.0 and 16.5 percentage points, respectively. These results suggest that combining textual and visual information benefits classification.
The modality ablation results here.
Third, we analyzed the test set across the four text-image relationship types. Supportive and Consistent examples account for 85% of the test set, while Contradictory and Complementary examples make up the remaining 15%. The latter two categories contain relatively few examples, and their results should therefore be interpreted with caution. Nevertheless, CATSF achieves 90.70%/94.12% accuracy on Contradictory/Complementary examples, showing promising performance even when textual and visual evidence conflict or must be jointly considered.
Performance across the four text-image relationship types is presented below.