acceptodds
Under review as a conference paper at ICLR 2027

CATSF: LEARNING ADAPTIVE REASONING FOR MULTIMODAL USER FEEDBACK CLASSIFICATION

Abstract

Android users generate large volumes of multimodal feedback, making manual triage impractical at scale and motivating automated classification. While multimodal large language models exhibit strong reasoning capabilities, high computational costs hinder their deployment, and the scarcity of high-quality task-specific data and reasoning supervision limits the adaptation of compact models. Reasoning-depth selection must also account for both text–image complexity and compact-model capabilities. To address these challenges, we introduce the Android User Feedback (AUF) dataset, a multimodal benchmark containing 3,750 expert-labeled entries, and propose CATSF, a Capability-Aware Teacher–Student Framework that separates learning to execute reasoning paths from learning to select them. In the first fine-tuning stage, a compact student learns basic and complex reasoning from teacher-generated trajectories. In the second stage, the same student learns adaptive path selection from teacher targets selected based on whether its first-stage basic predictions match the training labels. Experiments show that CATSF achieves a relative macro- improvement of up to 63.4% and an accuracy gain of up to 18.3 percentage points over the evaluated baselines. Across teacher–student configurations, CATSF reduces mean output token usage by up to 50.8% relative to matched, separately fine-tuned complex-reasoning baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Android User Feedback, Adaptive Reasoning Depth, Multimodal Interdependencies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sbSFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents CATSF, a capability-aware teacher-student framework for adaptive reasoning in multimodal user feedback classification, along with a new Android User Feedback (AUF) dataset containing 3,750 expert-annotated text-image examples across 21 categories. CATSF employs a two-stage fine-tuning strategy: first, a compact student model learns basic and complex reasoning trajectories generated by teacher MLLMs; second, it learns to adaptively select reasoning paths using supervision derived from its own classification correctness under basic reasoning. Experiments with two teacher models and two student models demonstrate improvements over the evaluated classification baselines.

Critical strengths

1.The paper addresses the practical challenge of balancing classification performance and computational efficiency in multimodal user feedback analysis. Considering both input complexity and student-model capability when selecting reasoning depth provides a reasonable motivation for the proposed approach. 2.CATSF introduces a concise two-stage fine-tuning framework that separates inference learning from adaptive inference path selection.

Critical weaknesses

1.Insufficient Motivation and Justification for the New AUF Dataset The motivation and necessity of introducing AUF are not sufficiently justified in the main paper. Existing Android user feedback and bug-report datasets, such as Android Apps and User Feedback [1] and the Android GUI-based bug-report dataset [2], are not adequately discussed or compared. Although these datasets differ from AUF in their tasks and modalities, the paper does not clearly establish what unique challenges AUF addresses or why a new benchmark is necessary. A more comprehensive discussion of existing resources and a clearer justification of AUF's distinctive contributions should be included in the main paper. [1]Grano, Giovanni, et al. "Android apps and user feedback: a dataset for software evolution and quality improvement." Proceedings of the 2nd ACM SIGSOFT international workshop on app market analytics. 2017. [2]Mahmud, Junayed, et al. "On using gui interaction data to improve text retrieval-based bug localization." Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 2024.

2.Limited Comparisons with Relevant Adaptive Reasoning Methods Although Table 2 compares CATSF with several multimodal classification baselines, including BLIP-2, Multimodal-CoT, and WisdoM these methods do not directly address adaptive reasoning-path selection. More closely related approaches, such as Thinkless [3], TON [4], and TATA [5], are discussed in Section 2.2 but are not included in the experimental comparisons. Comparisons with these methods or appropriately adapted variants under comparable backbone and training settings would better establish the effectiveness of the proposed approach. [3] Fang, Gongfan, Xinyin Ma, and Xinchao Wang. "Thinkless: Llm learns when to think." Advances in neural information processing systems 38 (2026): 151268-151295. [4] Wang, Jiaqi, et al. "Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models." Advances in Neural Information Processing Systems 38 (2026): 111196-111234. [5] Xu, Xin, et al. "Teaching LLMs According to Their Aptitude: Adaptive Switching Between CoT and TIR for Mathematical Problem Solving." The Third Conference on Parsimony and Learning (Proceedings Track). 2026.

Questions for authors
  1. What specific limitations of existing Android user feedback datasets motivate the construction of AUF? What distinctive challenges does AUF introduce compared with existing datasets?
  2. The anonymous repositories do not appear to provide the image files or retrieval URLs for AUF.
  3. Tables 3 and 6 demonstrate performance improvements after two-stage fine-tuning. How can the authors distinguish gains from adaptive reasoning-path selection from those resulting from additional supervised training?
  4. Why are closely related adaptive reasoning methods, such as Thinkless, TON, and TATA, not included in the experimental comparisons?
  5. Given that capability-aware reasoning-path selection is not specific to Android user feedback, can the authors evaluate CATSF on established multimodal benchmarks to demonstrate its generalizability?
  6. Why does adaptive reasoning achieve the highest Macro-F1 in only two of the four configurations in Table 3? Can the authors provide per-class performance results and text-only/image-only ablations, together with analyses across the four defined text-image relationship types?
2 comments

Q1 and Q2 We thank the reviewer for raising this concern. Existing Android feedback datasets lack support for multimodal classification: Android Apps and User Feedback [1] contains primarily textual feedback, while the GUI-based bug-report dataset [2] focuses on bug localization.

AUF addresses these gaps with 3,750 expert-labeled text–image pairs. It also introduces three challenges: unreliable user-provided labels, fine-grained classification across 21 technical categories requiring domain expertise, and heterogeneous text–image relationships requiring cross-modal reasoning.

Q3. We have approximately 1.6 GB of image files, hosted in the Releases section of the original GitHub repository. We have also added example data entries to examples/posts.json.

Q4. We conducted two controlled experiments to disentangle the effects of additional training and capability-aware selection. First, continuing Stage 1 SFT with the same number of updates achieved 81.50%, below CATSF (86.00%), suggesting that additional training alone does not explain the gains. Second, replacing capability-aware selection with random selection under the same Stage 2 training budget achieved 83.50%, compared with 86.00% for CATSF.

Q5. Our evaluation focuses on student-correctness-guided SFT for multimodal feedback classification. As discussed in Sec. 2.2, Thinkless uses RL for reasoning-length selection, TON combines thought-dropout SFT with GRPO, and TATA selects CoT/TIR demonstrations via anchor-set evaluation. In contrast, CATSF selects teacher targets based on the task-adapted student's basic-path correctness and learns selection through a second SFT stage, without RL or tool execution. Fair comparisons would require substantial pipeline adaptations and budget controls, potentially confounding method effects with implementation choices.

Q6. Although evaluated on Android user feedback classification, CATSF's capability-aware reasoning strategy is not inherently domain-specific. Applying it to other multimodal benchmarks would require adapting the reasoning paths and prompts. We leave such evaluations for future work.

Q7. The main reason is the class imbalance in reliable selection supervision. Some categories in the website are extremely underrepresented, such as GPS and Location. Our additional evaluation shows that Gemini-2.5-flash achieves only 0%, 0%, and 66.7% accuracy on these categories, respectively, compared with 50%, 50%, and 100% for GPT-4o-mini. Since CATSF retains only teacher trajectories with correct final predictions, fewer reliable examples from these minority classes are available for selection training under Gemini supervision. Consequently, the learned selection policy is more influenced by majority classes, potentially leading to less reliable reasoning-path choices for minority classes. This favors overall Accuracy but limits Macro-F1.

Q8. We conducted three additional experiments.

First, we evaluated classification accuracy across individual categories. The results show varying performance across categories, with relatively lower accuracy on Accessibility, Location, and Performance. These categories remain more challenging for the model.

Per-class accuracy results are presented below.

CategoryAccuracyCategoryAccuracyCategoryAccuracy
Camera97.8%Notifications94.4%Accessibility40.0%
Battery98.2%Security71.4%Backup&Restore80.0%
Other77.8%Malware85.7%Performance60.0%
Settings74.3%DeviceProtection85.7%Syncing100.0%
GooglePlayServices74.1%Privacy&Permissions66.7%ForgotPIN,Pattern,Passcode100.0%
Wi-Ficonnectivity81.8%Bluetooth100.0%GPS100.0%
Stability72.7%FindMyDevice100.0%Location50.0%

Second, we conducted text-only and image-only ablations using Qwen2.5-VL-7B, comparing them with the full CATSF model. The full model achieves 86.0% accuracy, outperforming the text-only and image-only variants by 6.0 and 16.5 percentage points, respectively. These results suggest that combining textual and visual information benefits classification.

The modality ablation results here.

Qwen2.5-VL-7BAccuracyMacro-F1
text-only80.0072.26
image-only69.5053.03
CATSF86.0080.40

Third, we analyzed the test set across the four text-image relationship types. Supportive and Consistent examples account for 85% of the test set, while Contradictory and Complementary examples make up the remaining 15%. The latter two categories contain relatively few examples, and their results should therefore be interpreted with caution. Nevertheless, CATSF achieves 90.70%/94.12% accuracy on Contradictory/Complementary examples, showing promising performance even when textual and visual evidence conflict or must be jointly considered.

Performance across the four text-image relationship types is presented below.

RelationNAccuracy
Supportive24185.06
Consistent9984.85
Contradictory4390.70
Complementary1794.12

Thank you for the additional experiments and clarifications. However, comparisons with closely related adaptive reasoning methods, such as Thinkless, TON, and TATA, remain necessary to demonstrate the effectiveness and competitiveness of CATSF. I will keep my score.

Reviewer_2SBpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores Android user feedback on multimodal large language models. To compensate for the scarcity of high-quality taskspecific data and reasoning supervision, the authors collected the Android User Feedback dataset, which contains 3,750 expert-labeled entries. In addition, the authors also introduced a capability-aware teacher–student framework that separates learning to execute reasoning paths from learning to select them. Experimental results show that the proposed method achieves performance improvements by a large margin.

Critical strengths
  1. The exploration of Android user feedback is interesting and meaningful. The proposed Android User Feedback dataset would be useful for future researchers.

  2. The authors evaluated both proprietary and open-source models on the proposed dataset and provided the dataset statistics.

  3. The authors conducted extensive evaluation and ablation studies to validate the key motivation of the proposed adaptive reasoning framework.

Critical weaknesses
  1. Although the authors provide dataset statistics for the proposed Android user feedback dataset, there are no data samples in the appendix. It is hard to know the details of the data.

  2. Figure 2 is so simple and mostly looks like an AI-generated image. Such an image should be polished to convey the key solution of the proposed capability-aware teacher–student framework.

  3. Some descriptions are hard to understand. For example, there is no explanation for the model "Qwen-2.5-VL(GPT)". Whether the Qwen-VL(GPT) denotes "Qwen-2.5-VL(GPT)"? In addition, what is "Qwen-VL-7B-Instruct fine-tuned via GPT-4o Mini dataset"? There are no details for this model.

Questions for authors

Basically, I think this paper is not up to the bar of ICLR. I have no questions for the authors. The critical weaknesses have been listed.

2 comments

Responses For Weakness:

1. Although the authors provide dataset statistics for the proposed Android user feedback dataset, there are no data samples in the appendix. It is hard to know the details of the data.

Thank you for highlighting presentation concerns. Figure 1 (in page 2) shows an AUF text-image example, and Appendix A.3.1 (in page 19) explains a dataset case through its textual and visual evidence. Appendix A.2 details data construction and annotation.

2. Figure 2 is so simple and mostly looks like an AI-generated image. Such an image should be polished to convey the key solution of the proposed capability-aware teacher–student framework.

Figure 2 depicts teacher-trajectory filtering, Stage-1 reasoning training, student-correctness-based target selection, Stage-2 selection training, and teacher-free inference. Please clarify which connection or component remains unclear.

3. Some descriptions are hard to understand. For example, there is no explanation for the model "Qwen-2.5-VL(GPT)". Whether the Qwen-VL(GPT) denotes "Qwen-2.5-VL(GPT)"? In addition, what is "Qwen-VL-7B-Instruct fine-tuned via GPT-4o Mini dataset"? There are no details for this model.

We acknowledge inconsistent shorthand. Qwen-2.5-VL(GPT) and Qwen-VL(GPT) refer to the same Qwen2.5-VL-7B-Instruct student trained through CATSF using GPT-4o-mini-generated AUF trajectories. “GPT-4o Mini dataset” denotes these teacher-generated targets, not a separate dataset (explained in the first paragraph of Section 4.1.2); their construction and training are described in Secs. 3.2–3.3. We respectfully ask the reviewer to reconsider the assessment in light of these clarifications and identify any remaining technical concerns.

Thanks to the authors' response. However, this response did not address my concerns.

  1. The data samples are not enough to represent the full dataset. The authors should refine this manuscript next time.

  2. In my opinion, Figure 2 is hard to understand the key idea. The authors should refine it and make it clearer.

  3. Thank you for the explanation.

Therefore, I intend to keep my initial rating.

Reviewer_pFujOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Android User Feedback (AUF) dataset, containing 3,750 expert-labeled text–image feedback entries across 21 categories, and proposes CATSF, a two-stage teacher–student framework. In Stage 1, the student learns basic and complex reasoning trajectories generated by teachers and filtered for correct final labels. In Stage 2, the first-stage student’s basic-reasoning correctness on training examples determines which teacher trajectory is used for further fine-tuning. At inference, the student selects a reasoning path and predicts the category without querying a teacher. The method achieves strong classification results on the 400-example AUF test set.

Critical strengths
  1. The task and dataset are valuable. The paper describes independent annotation by two annotators and a process for resolving disagreements. It reports a mean category-wise κ of 0.7253 (Appendix A.2, Table 7).
  2. The complete pipeline shows clear end-to-end gains. The GPT-supervised Qwen-VL student reaches 86.0% accuracy and 80.4 macro-F1, outperforming the evaluated baselines in Table 2. The full pipeline also outperforms both ablated variants in Table 6. The reported reduction in output tokens relative to fixed complex reasoning is useful, although it does not measure total deployment cost.dule.
Critical weaknesses
  1. Direct baselines with matched training budgets are missing, making the independent contribution of capability-aware selection unclear. The paper compares several classification methods but does not adequately compare simpler training procedures using the same pool of teacher trajectories. Useful controls would include continuing SFT on all trajectories filtered for correct final labels with a matched number of training tokens or updates, and randomly selecting an available basic or complex trajectory while matching the Stage 2 sample count and prompt. Because CATSF itself combines trajectory filtering with an additional training stage, the current results do not establish how much of the gain comes from the proposed selection rule.
  2. The ablations do not isolate the effect of the selection rule. “B&C w/o” removes Stage 1 training and changes the student whose predictions construct the Stage 2 dataset. “Adap w/o” also removes the Stage 2 data and training compute (Table 6). These experiments show that the complete pipeline outperforms the reduced variants, but they do not show that student-correctness-guided selection is better than random selection or ordinary continued SFT under a matched budget.
Questions for authors

For the same student checkpoint and test examples, how many examples fall into each of the four combinations of correct or incorrect basic and complex predictions? Among examples where basic reasoning fails but complex reasoning succeeds, how often does the final model select the complex path and correct the classification? A basic-path failure may reflect insufficient domain knowledge or visual perception rather than insufficient reasoning depth; the routing agreement in Table 9 cannot distinguish these cases. How many AUF test examples contain image evidence that adds information needed for classification beyond the text? How many exhibit complementary or contradictory text–image evidence, and how do the models perform on these subsets? If these relationships have not been annotated, could the authors specify annotation criteria and label the test set or a representative subset?

1 comment

Response For Weakness:

We thank you for pointing out that the original ablations did not fully isolate the effect of capability-aware selection from additional training. To address this, we conducted two controlled experiments. First, we continued Stage 1 SFT on all trajectories with correct final labels, using the same number of training updates. This baseline achieved 81.50%, below both CATSF (86.00%) and the original Adap w/o variant (82.50%), indicating that additional SFT alone does not explain the improvement.

Second, we replaced capability-aware selection with random selection between available basic and complex trajectories, while matching the Stage 2 sample count, adaptive prompt, and training budget. This variant achieved 83.50%, compared with 82.50% for Adap w/o and 86.00% for CATSF. This suggests that adaptive-prompt fine-tuning provides some benefit, while capability-aware selection further improves performance over random selection. These results suggest that capability-aware selection contributes to the performance gains of CATSF beyond the benefits of additional SFT and adaptive-prompt fine-tuning. We will include both experiments in the revised ablation study.

Responses For Questions:

Q1. We evaluate the same Stage-1 student checkpoint on the same 400 test examples using both basic and complex reasoning prompts. The results are summarized below:

CaseCountProportion (%)
1) Basic ✓ & Complex ✓31679.00%
2) Basic ✓ & Complex ✗133.25%
3) Basic ✗ & Complex ✓194.75%
4) Basic ✗ & Complex ✗5213.00%

These results provide two important observations. First, the student maintains comparable basic and complex reasoning capabilities to those of the separately fine-tuned variants reported in Table 3, suggesting that learning both reasoning paths does not substantially compromise either capability. Second, if we assume an oracle that always selects the correct prediction whenever either reasoning path succeeds, the theoretical upper-bound accuracy would be 87.00% (316 + 13 + 19 out of 400). Notably, CATSF achieves 86.00% accuracy, coming within just 1 percentage point of this oracle upper bound. This demonstrates the effectiveness of CATSF in learning to adaptively select between basic and complex reasoning paths.

Q2. We thank the reviewer for this important observation. Among the 19 cases where basic reasoning fails but complex reasoning succeeds, the adaptive model selects the complex path in 15 cases (78.9%), correctly classifying 12 of them (80.0%). This corresponds to a recovery rate of 63.2% (12/19). We acknowledge that basic-path failures may also stem from limitations in domain knowledge or visual perception. Indeed, both paths fail on 52 cases, suggesting that deeper reasoning alone may not resolve all classification errors. We will clarify this distinction in the revised manuscript.

Q3, Q4, and Q5 We sincerely thank the reviewer for raising these important questions about the role of visual evidence and the interactions between text and images in AUF.

To assess the contribution of visual information, we conducted an additional text-only experiment using Qwen2.5-VL-7B, which achieved 80% accuracy. This is below the approximately 87% theoretical upper bound estimated in our previous analysis, suggesting the potential benefit of visual evidence. For reference, the text-only Qwen2.5-7B baseline achieves 76.3% accuracy.

Furthermore, based on the four text–image relationship definitions in Appendix A.3, we conducted an analysis of the existing 400 test examples. The results are summarized below:

RelationNAccuracy (%)
Supportive241 (60.25%)85.06%
Consistent99 (24.75%)84.85%
Contradictory43 (10.75%)90.70%
Complementary17 (4.25%)94.12%

In particular, 60 examples (15%) exhibit either complementary or contradictory text–image relationships. Among them, the 17 complementary cases are especially important, as neither modality alone provides sufficient information for classification, whereas their combination reveals the underlying issue. Despite the relatively small number of such cases, our CATSF-trained model achieves 94.12% accuracy on complementary examples and 90.70% on contradictory examples, providing some initial insights into its performance on these types of cross-modal relationships.

These findings suggest that multimodal understanding requires not only identifying consistent evidence but also integrating complementary information and resolving cross-modal conflicts, providing further insights into CATSF's cross-modal reasoning.

We will incorporate these analyses, annotation criteria, and relationship distributions into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.