acceptodds
Under review as a conference paper at ICLR 2027

GLANCING, FOCUSING, REASONING: A COGNITION-INSPIRED FRAMEWORK FOR FINE-GRAINED FOOD RECOGNITION

Abstract

Since ancient times, understanding food has been essential to human survival, development, and cultural formation, relying on visual perception and reasoning grounded in prior knowledge. Currently, multimodal large language models (MLLMs) exhibit remarkable capabilities in visual-semantic understanding. However, enabling MLLMs to understand fine-grained food categories poses three food-specific challenges: discovering non-part-aligned discriminative cues under interference from shared superclass patterns, recognizing ingredients under inter-ingredient similarity and intra-ingredient appearance variation, and reasoning over ingredient roles and relations. Inspired by cognitive science, we decompose food understanding into three progressive cognitive actions to address these challenges: glancing establishes coarse food context for discriminative cue discovery; focusing localizes regions to recognize ingredient identity and state; and reasoning integrates high-level semantics such as cooking methods, ingredient roles, and inter-ingredient relations. Accordingly, we propose GFRFood, a cognition-inspired MLLM framework for fine-grained food recognition (FGFR) that instantiates these three cognitive actions through two complementary modules. Cognition-Inspired Progressive Semantic Reasoning (CIPSR) generates food attributes by progressively integrating discriminative ingredient cues and high-level semantics; Taxonomy-Conditioned Semantic State-Space Model (TC-S3M) uses a selective state-space model to refine fine-grained category understanding. The two modules model visual and taxonomy semantics, respectively, and jointly enhance the MLLM's fine-grained understanding. Extensive evaluations demonstrate that GFRFood outperforms general, fine-grained, and food-specific MLLMs, achieving state-of-the-art performance. Ablation studies and visualizations demonstrate the advantages of GFRFood in attribute quality and taxonomic understanding. Dataset and code will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fine-Grained Food Recognition, Multimodal Large Language Models, Cognition-Inspired Semantic Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MyAYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GFRFood, a cognition-inspired framework for fine-grained food recognition using multimodal large language models (MLLMs). The proposed GFRFood framework first introduces Cognition-Inspired Progressive Semantic Reasoning (CIPSR) by employing multiple LLM-based agents to generate discriminative food attributes through three progressive stages: glancing, focusing, and reasoning. In addition, it further includes Taxonomy-Conditioned Semantic State-Space Model (TC-S3M) models hierarchical food categories through selective state transitions, providing complementary category predictions that are fused with the MLLM outputs. The authors also construct FoodFG, a benchmark curated from several existing datasets using MCQ. Experiments show that GFRFood outperforms general-purpose and food-specific MLLMs on MCQ for fine-grained food recognition.

Critical strengths
  1. The three challenges identified in the introduction are reasonably motivated, and the examples in Figure 1 help illustrate why conventional visual-semantic understanding can be insufficient.
  2. The combination of ingredient recognition, visual attribute generation, and hierarchical category modeling is intuitively meaningful for this task.
  3. The paper writing is easy to follow and the figures are illustrative and helpful.
Critical weaknesses
  • A key concern is that the final performance improvement is not solely attributable to the proposed progressive reasoning mechanism. According to Table 5, CIPSR achieves 78.31% R-acc, while combining it with TC-S3M increases accuracy to 90.24%. TC-S3M itself is a separately trained supervised classifier operating on visual features and hierarchical category labels.
  • The proposed method is much more complex compared with existing methods, combining several components, such as attribute-based fine-grained recognition, multi-agent verification, hierarchical classification and selective state-space models. These components are not new in general, though the proposed method composes these into a framework.
  • The main experiments use four-way MCQ on FoodFG, with training and test images drawn from the same set of food categories. While this is a reasonable evaluation protocol, it does not fully establish generalization to more realistic recognition scenarios involving unseen dishes, varying candidate sets, or unconstrained category prediction. It would be good to see more generalization experiments except for the zero-shot evaluation in Table 4.
  • Adding some results of the latest closed-source models for reference would be helpful, such as GPT, Gemini and Claude. Please list the computational cost comparison with the existing methods as well.
  • It would be great to add more details about human annotations and review, as well as more stats for this process.
Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Part 1. Core module contributions and novelty

Rebuttal:

We sincerely thank the reviewer for the constructive suggestions. We address each concern in detail below.

1. Contributions of CIPSR, TC-S3M, and supervised classification

GFRFood's gains come from CIPSR and from both taxonomy semantic modeling and supervised classification in TC-S3M.

1) Contributions of CIPSR and TC-S3M. In revised Table 5 (Section 4.3), TC-S3M without MLLM fusion achieves 74.37% in R-Acc. Fusing TC-S3M with instruction-tuned Qwen3-VL improves R-Acc. to 81.54% (+10.79%). CIPSR with instruction tuning improves R-Acc. from 70.75% to 78.31% (+7.56%) over the instruction-tuned baseline. With TC-S3M fusion, adding CIPSR improves R-Acc. from 81.54% to 90.24% (+8.70%).

Table 5: Effects of CIPSR and TC-S3M (%).

\hline \text{Configuration}&\text{R-Acc.}&\text{Rot-Acc.}&\text{Rot-Hit}\\\\ \hline \text{TC-S3M only}&74.37&74.37&74.37\\\\ \hline \text{Baseline}&57.21&49.40&57.11\\\\ \quad \text{+ TC-S3M}&66.89&64.24&66.75\\\\ \hline \text{I.T.-only}&70.75&70.06&70.82\\\\ \quad \text{+ TC-S3M}&81.54&81.02&81.78\\\\ \hline \text{CIPSR + I.T.}&78.31&76.65&78.32\\\\ \quad \text{+ TC-S3M (Ours)}&\mathbf{90.24}&\mathbf{89.72}&\mathbf{90.23}\\\\ \hline \end{array}$$ **2) Contributions of taxonomy modeling and supervised classification in TC-S3M.** Revised Table 7 (Section 4.3) assesses whether TC-S3M benefits from taxonomy modeling beyond classification supervision. Specifically, we compare TC-S3M with three supervised classifiers trained on the same frozen Qwen3-VL visual features, selecting each classifier's best-performing fusion weight $w$. Flat-Linear uses a fully connected layer, Flat-MLP uses residual MLP blocks, and Hier-MLP adds three auxiliary taxonomy heads to Flat-MLP. With the CIPSR and instruction tuning backbone, Flat-Linear fusion improves R-Acc. from 78.31% to 86.72% (+8.41%), while TC-S3M fusion improves it from 78.31% to 90.24% (+11.93%) and outperforms Flat-Linear fusion by 3.52%. Therefore, supervised classification improves FGFR, and TC-S3M's further gains support the benefit of modeling food taxonomy semantics. Appendix A.4.4.3 (Table S10) further reports classifier architectures, computational costs, and classifier-only results. > Table 7: Comparison of TC-S3M with different supervised classifiers on Relaxed Accuracy (%). $$\begin{array}{lrrrrr} \hline \text{Model}&\text{No fusion}&\text{Flat-Linear}&\text{Flat-MLP}&\text{Hier-MLP}&\text{TC-S3M}\\\\ \hline \text{Classifier only}&--&71.46&70.76&71.30&\mathbf{74.37}\\\\ \hline \text{Baseline}&57.21&64.87&64.09&64.98&\mathbf{66.89}\\\\ \text{Baseline + I.T.}&70.75&77.90&76.39&77.93&\mathbf{81.54}\\\\ \text{CIPSR + I.T.}&78.31&86.72&85.35&86.94&\mathbf{90.24}\\\\ \hline \end{array}$$ **2. Novelty analysis** Our main contribution is a set of cognition-inspired mechanisms for MLLMs to capture fine-grained food semantics more accurately and comprehensively. We use attribute-based fine-grained recognition, multi-agent verification, and hierarchical classification to realize three complementary designs. **1) Cognition-inspired visual food understanding.** Inspired by how humans understand food, CIPSR extends attribute-based recognition by decomposing food understanding into Glancing, Focusing, and Reasoning. Glancing establishes global context to discover discriminative cues. Focusing identifies ingredients and their states from local visual evidence. Reasoning integrates visual cues with cooking methods and ingredient relations. This mechanism guides MLLMs to recover missed discriminative cues and disambiguate ingredient identities and states. In revised Table 4 (Section 4.3), CIPSR outperforms the attribute-based method FineDefics with Self-Refine by 4.29% in R-Acc. at comparable annotation cost. **2) Reliable cognitive attribute construction.** Multi-agent orchestration improves attribute reliability by verifying and correcting MLLMs' ingredient recognition and semantic interpretation. Decoupling ingredient identity and state preserves ingredient visual–semantic correspondence under various ingredient forms. The prior-guided and prior-free Extractor branches capture complementary category prior semantics and image-specific visual cues. To mitigate confirmation bias, counterfactual verification assumes each ingredient candidate is incorrect and revises its identity or state if an alternative has stronger local visual support. Appendix A.4.2 (Tables S3–S4) evaluates these agent contributions. **3) Coarse-to-fine food taxonomy modeling.** TC-S3M follows the progression of human food categorization from coarse food clusters through ingredient families to processing form. TC-S3M uses ancestor taxonomy semantics to refine category features from coarse to fine. Appendix A.4.4 (Tables S8–S10) shows that TC-S3M outperforms flat classifiers and conventional hierarchical classification models.
AuthorsOpenReview ↗

Part 2. Generalization, closed-source models, costs, and human review

3. Generalization across datasets, varying candidate sets, and unconstrained category prediction

We extend our evaluation with cross-dataset experiments, MCQs with larger candidate sets, and unconstrained generation.

1) Cross-dataset generalization. In Appendix A.4.7 (Table S16), we evaluate the cross-dataset generalization of our method on DiningBench-4, DiningBench-8, CNFOOD-241, and FoodX-251 without target-dataset fine-tuning. Compared with Qwen3-VL, CIPSR achieves R-Acc. of 67.57% (+7.24%), 65.93% (+7.30%), 76.48% (+5.17%), and 80.75% (+6.75%), respectively.

Table S16: R-Acc. (%) on external food benchmarks.

\hline \text{Model}&\text{DiningBench-4}&\text{DiningBench-8}&\text{CNFOOD-241}&\text{FoodX-251}\\\\ \hline \text{Qwen3-VL}&60.33&58.63&71.31&74.00\\\\ \text{Qwen3-VL + I.T.}&55.93&51.04&71.93&68.75\\\\ \text{CIPSR}&\mathbf{67.57}&\mathbf{65.93}&76.48&\mathbf{80.75}\\\\ \text{CIPSR + I.T.}&61.21&59.76&\mathbf{76.91}&78.60\\\\ \hline \end{array}$$ **2) Larger MCQ candidate sets.** At candidate set sizes of 16 and 64 (Appendix A.4.1, Table S2), GFRFood outperforms the instruction-tuned baseline by 18.23% and 18.50% in R-Acc., respectively. Its advantage holds with larger candidate sets. > Table S2: Relaxed accuracy (%) on text-based MCQs across candidate set sizes. $$\begin{array}{lrrr} \hline \text{Model}&K=4&K=16&K=64\\\\ \hline \text{Qwen3-VL}&57.21&42.15&31.55\\\\ \text{Qwen3-VL + I.T.}&70.75&69.93&67.61\\\\ \text{CIPSR + I.T.}&78.31&76.61&75.02\\\\ \text{GFRFood}&\mathbf{90.24}&\mathbf{88.16}&\mathbf{86.11}\\\\ \hline \end{array}$$ **3) Unconstrained category prediction.** Revised Table 6 (Section 4.3) evaluates all 1,622 categories. The MLLM generates a category name using only the image and a fixed question, without a category list or TC-S3M fusion. We match the category name in the first 20 generated tokens to the ground truth using semantic similarity. CIPSR with instruction tuning improves Top-1 accuracy from 65.61% to 77.91% (+12.30%) over the instruction-tuned baseline. > Table 6: Top-1 accuracy (%) for unconstrained category prediction. $$\begin{array}{lr} \hline \text{Model}&\text{Top-1 Acc.}\\\\ \hline \text{Baseline}&45.84\\\\ \text{Baseline + I.T.}&65.61\\\\ \text{CIPSR}&66.85\\\\ \text{CIPSR + I.T.}&\mathbf{77.91}\\\\ \hline \end{array}$$ **4. Closed-source model results and computational cost** Revised Table 1 (Section 4.2) adds three closed-source models. GFRFood surpasses GPT-5.6 SOL by 20.86% and 21.01% in R-Acc. on text-based and prototype-based MCQs, respectively. > Table 1: Relaxed accuracy (%) of GFRFood and closed-source MLLMs on FoodFG. $$\begin{array}{lrr} \hline \text{Model}&\text{Text-based MCQs}&\text{Prototype-based MCQs}\\\\ \hline \text{GPT-5.6 SOL}&69.38&71.17\\\\ \text{Gemini 3.1}&67.85&70.06\\\\ \text{Claude Sonnet 4.6}&62.27&64.54\\\\ \text{GFRFood}&\mathbf{90.24}&\mathbf{92.18}\\\\ \hline \end{array}$$ We report costs for offline attribute annotation and online inference. **1) Offline attribute annotation.** Revised Table 4 (Section 4.3) compares the annotation costs of CIPSR and existing attribute generation methods. CIPSR’s offline annotation costs USD 728.47 (USD 4.34 per 1,000 images). At comparable cost, CIPSR outperforms FineDefics with Self-Refine by 4.29% in R-Acc. > Table 4: Annotation costs and relaxed accuracy (%) of attribute generation methods. $$\begin{array}{lrrr} \hline \text{Method}&\text{Tokens/Img.}&\text{Cost/1K (USD)}&\text{R-Acc.}\\\\ \hline \text{Direct}&\text{2,013}&0.46&70.56\\\\ \text{FineDefics}&\text{3,592}&1.26&70.89\\\\ \text{FineDefics + Self-Refine}&\text{10,872}&4.24&74.02\\\\ \text{CIPSR (Ours)}&\text{11,205}&4.34&\mathbf{78.31}\\\\ \hline \end{array}$$ **2) Online inference.** For inference-cost comparisons, fine-tuned Qwen3-VL is the backbone reference. CIPSR runs offline, and TC-S3M adds 0.068 GFLOPs per image. **5. Human annotation procedures and review statistics** Appendix A.1 details human review. **1) Category standardization.** Human review removes 457 coarse-grained categories when fine-grained subclasses coexist. **2) Image screening.** Human review removes 1,792 of 2,076 images near the label screening thresholds. **3) Deduplication.** Human review removes 615 of 747 images near the similarity thresholds. **4) Manual review.** One annotator re-examines category definitions and removes 12 categories containing 2,214 images, leaving 284,935 images across 1,622 categories. **5) Final audit.** One annotator checks image quality and image–label consistency in 1,000 randomly sampled images (seed 42). Of these, 947 pass both criteria (94.7%). Appendix A.4.2 reports a gold set of 4,866 images, with three per category. One annotator labels ingredient identities, visible states, and bounding boxes, followed by spot checks.
Reviewer_eaBmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GFRFood, a cognition-inspired framework for fine-grained food recognition that decomposes food understanding into three progressive stages: Glancing, Focusing, and Reasoning. The framework consists of two complementary components: CIPSR, a multi-agent pipeline that generates discriminative ingredient- and semantic-level attributes for model fine-tuning, and TC-S3M, a taxonomy-conditioned state-space model that progressively incorporates coarse-to-fine food taxonomy information. The paper also constructs FoodFG, a large-scale benchmark containing 284935 images across 1622 fine-grained food categories, and evaluate the framework on both supervised and zero-shot settings.

Critical strengths
  1. The paper Identifies practically important challenges in fine-grained food recognition and organizes them into a coherent progression from global context and ingredient-level evidence to higher-level relational semantics, providing an intuitive formulation of the task.
  2. The paper proposes a framework that combines food-specific attribute supervision with explicit taxonomy modeling. The two components capture complementary visual-semantic and hierarchical information, and the extensive ablations in the main manuscript and appendix provide useful evidence for their effectiveness.
Critical weaknesses
  1. The evaluation setting is narrow. FGFR evaluation relies on four-way MCQs despite FoodFG containing 1,622 fine-grained categories. This setting mainly measures discrimination among a small, predefined candidate set rather than full-label-space recognition, and so the reported gains may not fully reflect performance in large-scale FGFR.
  2. The major concern is train-inference mismatch in TC-S3M. Specifically, TC-S3M uses ground-truth ancestors during training but predicted ancestors during inference, and so errors at upper taxonomy levels may propagate to deeper predictions. That is to say, the reported per-level accuracies do not isolate this effect.
Questions for authors

Please address the weaknesses. Besides, some more questions are as follows.

  1. CIPSR relies on a multi-agent GPT-5 annotation pipeline, but the trade-off between annotation cost and recognition performance is unclear. Please report the annotation cost and evaluate performance under different annotation budgets to better assess the efficiency and scalability of CIPSR.
  2. Is Glancing-Focusing-Reasoning explicitly performed at inference time? CIPSR appears to generate offline attributes for fine-tuning, and so it remains unclear whether the deployed model actually follows the proposed cognitive process or mainly benefits from the structured synthetic supervision.
  3. The sensitivity to key hyperparameters is not analyzed. The manuscript fixes wl= {0.3,0.3,0.4} and w=0.3, respectively, without explaining how they were selected or whether the reported gains are robust to these choices.
2 comments
AuthorsOpenReview ↗

Part 1: Full-label-space recognition and ancestor error propagation

Rebuttal:

We sincerely thank the reviewer for the constructive suggestions. We address each concern in detail below.

1. Full-label-space recognition

Revised Table 6 (Section 4.3) evaluates the effectiveness of our method for full-label-space recognition over all 1,622 categories using two protocols. 1) Closed-set linear probing. We train linear classifiers on frozen LLMs' last input-token features using the prompt “Identify the food shown in the image.” 2) Unconstrained generation. The MLLM generates a category name using only the image and a fixed question, without a category list or TC-S3M fusion. We match the category name in the first 20 generated tokens to the ground truth using semantic similarity.

In closed-set probing, CIPSR with TC-S3M improves Top-1 accuracy from 75.71% to 88.98% (+13.27%) compared with Baseline. In unconstrained generation, CIPSR with instruction tuning improves Top-1 accuracy from 65.61% to 77.91% (+12.30%) over the instruction-tuned baseline. Both evaluations demonstrate the effectiveness of our method for full-label-space recognition.

Table 6: Full-label-space Top-1 accuracy (%).

\hline \text{Model}&\text{Probe}&\text{Generation}\\\\ \hline \text{Baseline}&75.71&45.84\\\\ \text{Baseline + I.T.}&73.68&65.61\\\\ \text{Baseline + TC-S3M}&83.20&--\\\\ \text{Baseline + I.T. + TC-S3M}&81.60&--\\\\ \hline \text{CIPSR}&79.50&66.85\\\\ \text{CIPSR + I.T.}&77.42&\mathbf{77.91}\\\\ \hline \text{CIPSR + TC-S3M}&\mathbf{88.98}&--\\\\ \text{CIPSR + I.T. + TC-S3M}&86.59&--\\\\ \hline \end{array}$$ To assess GFRFood more broadly, we further evaluate our method in two sets of experiments. **1) Larger MCQ candidate sets.** Appendix A.4.1 (Table S2) reports results with 16 and 64 candidates. GFRFood outperforms the instruction-tuned baseline by 18.23% and 18.50% in R-Acc., respectively. **2) Cross-dataset generalization.** In Appendix A.4.7 (Table S16), we evaluate the cross-dataset generalization of our method on DiningBench-4, DiningBench-8, CNFOOD-241, and FoodX-251 without target-dataset fine-tuning. Compared with Qwen3-VL, CIPSR achieves R-Acc. of 67.57% (+7.24%), 65.93% (+7.30%), 76.48% (+5.17%), and 80.75% (+6.75%), respectively. > Table S16: R-Acc. (%) on external food benchmarks. $$\begin{array}{lrrrr} \hline \text{Model}&\text{DiningBench-4}&\text{DiningBench-8}&\text{CNFOOD-241}&\text{FoodX-251}\\\\ \hline \text{Qwen3-VL}&60.33&58.63&71.31&74.00\\\\ \text{Qwen3-VL + I.T.}&55.93&51.04&71.93&68.75\\\\ \text{CIPSR}&\mathbf{67.57}&\mathbf{65.93}&76.48&\mathbf{80.75}\\\\ \text{CIPSR + I.T.}&61.21&59.76&\mathbf{76.91}&78.60\\\\ \hline \end{array}$$ In summary, we assess our method through full-label-space recognition, larger MCQ candidate sets, and cross-dataset generalization. **2. Training–inference mismatch and error propagation** Unreliable early predictions may slow convergence (Bengio et al., 2015) and provide inconsistent parent context for selective state updates at deeper taxonomy levels. We therefore use ground-truth ancestors to provide semantically consistent context, and then switch to predicted ancestors in the final epoch to reduce the discrepancy between training and inference. In revised Table S11 (Appendix A.4.4.4), all training variants use predicted ancestors at inference. Our strategy outperforms training on predicted ancestors in L1–L3 accuracy and improves MCQ R-Acc. from 72.62% to 74.37% (+1.75%). > Table S11: Effects of training-time ancestor conditioning on TC-S3M (%). $$\begin{array}{lcccc} \hline \text{Training condition}&\text{L1 Acc.}&\text{L2 Acc.}&\text{L3 Acc.}&\text{MCQ R-Acc.}\\\\ \hline \text{Predicted ancestors}&91.58&87.06&81.35&72.62\\\\ \text{Ground-truth L1 only}&91.76&89.25&83.60&73.64\\\\ \text{Ground truth until final epoch (Ours)}&\mathbf{93.48}&\mathbf{90.11}&\mathbf{85.86}&\mathbf{74.37}\\\\ \hline \end{array}$$ Table S12 tests error propagation by varying only inference-time ancestor inputs with the checkpoint fixed. Ground-truth ancestors improve L2 and L3 accuracy, with a smaller gain of 1.23% in MCQ R-Acc. The fine-grained classifier combines the visual embedding $z_i$ (Eq. 4) with features from all three taxonomy levels, which may reduce the effect of ancestor errors on the final prediction. > Table S12: Effects of inference-time ancestor conditioning on TC-S3M (%). $$\begin{array}{lcccc} \hline \text{Inference condition}&\text{L1 Acc.}&\text{L2 Acc.}&\text{L3 Acc.}&\text{MCQ R-Acc.}\\\\ \hline \text{Predicted ancestors}&93.48&90.11&85.86&74.37\\\\ \text{Ground-truth L1 only}&93.48&\mathbf{93.04}&87.42&74.97\\\\ \text{Ground-truth ancestors}&93.48&\mathbf{93.04}&\mathbf{90.74}&\mathbf{75.60}\\\\ \hline \end{array}$$ **Reference.** Bengio et.al. (2015). Scheduled sampling for sequence prediction with recurrent neural networks. *Advances in Neural Information Processing Systems*, 28.
AuthorsOpenReview ↗

Part 2: Annotation cost, offline CIPSR attributes generation, and hyperparameter sensitivity

3. Annotation cost and accuracy under different budgets

We report annotation costs and evaluate recognition at different budgets.

1) Annotation cost and fair comparison. Revised Table S5 (Appendix A.4.3) reports CIPSR's API token usage and cost by stage. CIPSR costs USD 728.47 in total (USD 4.34 per 1,000 images). Appendix A.4.3 (Table S6) reports API calls, tokens, and costs by agent.

Table S5: API token usage and computational cost of CIPSR by stage.

\hline \text{Stage}&\text{Tokens (M)}&\text{Cost (USD)}&\text{Cost/1K (USD)}\\\\ \hline \text{Glancing}&23.18&9.27&0.055\\\\ \text{Focusing}&708.43&304.58&1.815\\\\ \text{Reasoning}&1149.26&414.62&2.470\\\\ \hline \text{Total}&1880.87&\mathbf{728.47}&\mathbf{4.340}\\\\ \hline \end{array}$$ In revised Table 4 (Section 4.3), we augment FineDefics with Self-Refine to assess whether additional computation explains CIPSR's gains. This baseline applies three-stage self-verification to assess visual support, coverage, and consistency at comparable cost. CIPSR outperforms this baseline by 4.29% in R-Acc. at comparable annotation cost (USD 4.34 vs. USD 4.24 per 1,000 images), demonstrating the effectiveness of CIPSR's structured glancing–focusing–reasoning pipeline. **2) Performance under different annotation budgets.** Revised Table S7 (Appendix A.4.3) assesses CIPSR's efficiency and scalability by varying the number of annotated images. At 25% and 50% of the full-budget image count, CIPSR improves R-Acc. over the instruction-tuned baseline to 74.68% (+3.93%) and 76.12% (+5.37%), respectively, indicating that CIPSR provides substantial gains even with limited annotation. > Table S7: Effects of CIPSR annotation budgets on text-based MCQ performance (%). $$\begin{array}{lrrr} \hline \text{Annotation budget}&\text{R-Acc.}&\text{Rot-Acc.}&\text{Rot-Hit}\\\\ \hline 0\\%&70.75&70.06&70.82\\\\ 25\\%&74.68&71.97&74.75\\\\ 50\\%&76.12&73.41&76.24\\\\ 100\\%&\mathbf{78.31}&\mathbf{76.65}&\mathbf{78.32}\\\\ \hline \end{array}$$ **4. How does the MLLM benefit from Glancing–Focusing–Reasoning?** CIPSR uses Glancing–Focusing–Reasoning only to construct training attributes offline. Through cross-modal alignment, these attributes improve the MLLM's ability to distinguish fine-grained food categories. At inference, the MLLM predicts categories without executing Glancing–Focusing–Reasoning. Inspired by human food understanding, CIPSR constructs discriminative attributes that describe ingredient identities and states, cooking methods, and ingredient relations. In Section 3.3, we align image and category-name representations with the corresponding attribute representations in the LLM's feature space. The attributes serve as a semantic bridge, linking visual cues to the fine-grained categories they distinguish. Instruction tuning then trains the MLLM to use these associations for category prediction. At inference, the MLLM can use the learned associations among visual cues, discriminative food attributes, and category semantics to predict fine-grained categories without explicitly executing the Glancing–Focusing–Reasoning stages. **5. Selection and sensitivity of fusion and taxonomy loss weights** We selected both sets of weights by grid search over the configurations in Appendix A.4.5. **1) Fusion weight** $w$. Revised Table S13 reports sensitivity to the fusion weight $w$. GFRFood achieves the highest R-Acc. of 90.24% at $w=0.3$. > Table S13: Sensitivity to the fusion weight $w$ (%). $$\begin{array}{lccc} \hline w&\text{R-Acc.}&\text{Rot-Acc.}&\text{Rot-Hit}\\\\ \hline 0.0&78.31&76.65&78.32\\\\ 0.2&89.18&86.54&89.18\\\\ 0.3&90.24&89.72&90.23\\\\ 0.5&90.16&89.45&90.05\\\\ 0.7&89.46&88.89&89.43\\\\ 1.0&74.37&74.37&74.37\\\\ \hline \end{array}$$ **2) Taxonomy loss weights** $\boldsymbol{\omega}$. Revised Table S14 reports sensitivity to the taxonomy loss weights $\boldsymbol{\omega}$. GFRFood achieves the highest fused R-Acc. of 90.24% at $\boldsymbol{\omega}=(0.3,0.3,0.4)$. > Table S14: Sensitivity to taxonomy loss weights $\boldsymbol{\omega}$ (%). $$\begin{array}{lccccc} \hline \boldsymbol{\omega}&\text{L1 Acc.}&\text{L2 Acc.}&\text{L3 Acc.}&\text{MCQ-Acc.}&\text{Fusion R-Acc.}\\\\ \hline (0.5,0.3,0.2)&\mathbf{93.63}&89.03&84.74&73.88&89.67\\\\ (0.2,0.6,0.2)&92.37&89.42&84.77&73.75&89.66\\\\ (0.2,0.2,0.6)&92.54&89.28&85.35&74.02&89.68\\\\ (0.3,0.3,0.4)&93.48&\mathbf{90.11}&\mathbf{85.86}&\mathbf{74.37}&\mathbf{90.24}\\\\ \hline \end{array}$$ R-Acc. ranges from 89.18% to 90.24% with fusion and from 89.66% to 90.24% across taxonomy loss weights, indicating that the gains are robust.
Reviewer_xfPxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GFRFood for fine-grained food recognition. CIPSR uses GPT-5 agents to generate and verify ingredient-centered attributes for attribute-alignment fine-tuning of Qwen3-VL-8B, followed by instruction tuning. TC-S3M learns a taxonomy-conditioned classifier from frozen visual features and combines its scores with those of the MLLM at inference.

The authors also introduce FoodFG, comprising 284,935 images from 1,622 categories. On text-based four-option MCQs, GFRFood achieves 90.24% R-Acc., compared with 71.25% for Fine-R1†, the strongest evaluated baseline.

Critical strengths
  1. CIPSR improves R-Acc. by 7.56 percentage points over instruction tuning alone and by 7.42 points over FineDefics (Tables 2–3). The 4.72-point gain on held-out categories is particularly encouraging, as it provides evidence of generalization beyond the fine-tuning classes (Table 4).
  2. The ablations support the attribute-generation pipeline. Verification and correction improve ingredient Macro F1 from 79.11 to 93.40 (Table S2), while full attributes outperform ingredient names alone, increasing R-Acc. from 72.52% to 78.31% (Table S3).
Critical weaknesses
  1. The current comparisons do not isolate the benefits of taxonomy supervision over a flat classifier or establish the contribution of CIPSR when classifier fusion is held fixed. Adding TC-S3M increases R-Acc. from 78.31% to 90.24% (Table 5), but the headline baselines do not use an additional classifier. Table S5 supports TC-S3M over the MLP and Transformer taxonomy variants (90.24% versus 84.41% and 86.60%). However, it does not include a flat classifier without taxonomy supervision. Instruction-tuned Qwen3-VL with the same TC-S3M fusion is also missing. These controls are needed to assess the value of taxonomy modeling and the contribution of CIPSR under matched fusion settings.
  2. The annotation settings for Direct and FineDefics are unclear (Table 3). GPT-5 is specified for CIPSR, but the models used to generate the baseline attributes are not identified. The stated replacement of the FineDefics backbone with Qwen3-VL does not resolve this ambiguity. The Direct prompt also includes a class prior, without establishing whether comparable category information is available to all methods. Differences in annotation models, prior information, or generation budgets could therefore contribute to the reported gains.

I lean toward weak rejection. The attribute-supervision gains and held-out-category results are promising, and Table S5 supports the proposed taxonomy module over the tested alternatives. However, the missing classifier controls and unclear annotation settings limit attribution of the gains to the proposed components. Matched comparisons and clarification of these settings could change my assessment. Reporting the GPT-5 annotation cost would also help assess practical feasibility.

Questions for authors
  1. How does a flat classifier without taxonomy supervision compare with TC-S3M when both use the same frozen visual features and are fused with the same CIPSR-trained MLLM?

  2. How does instruction-tuned Qwen3-VL perform with TC-S3M fusion? What is TC-S3M’s accuracy without MLLM fusion? The “TC-S3M” row in Table 5 reports fusion with untuned Qwen3-VL.

  3. Which annotation models, category information, and generation budgets were used for Direct, FineDefics, and CIPSR? What was the annotation cost of CIPSR?

1 comment
AuthorsOpenReview ↗

Added comparisons of TC-S3M with supervised classifiers, fusion ablations with instruction-tuned Qwen3-VL, and comparisons of attribute generation methods in terms of annotation models, category information, and generation budgets.

Rebuttal:

We sincerely thank the reviewer for the constructive suggestions. We address each concern in detail below.

1. Taxonomy modeling beyond classification supervision

Revised Table 7 (Section 4.3) assesses whether TC-S3M benefits from taxonomy modeling beyond classification supervision. Specifically, we compare TC-S3M with three supervised classifiers trained on the same frozen Qwen3-VL visual features, selecting each classifier's best-performing fusion weight ww. Flat-Linear uses a fully connected layer, Flat-MLP uses residual MLP blocks, and Hier-MLP adds three auxiliary taxonomy heads to Flat-MLP.

With the CIPSR and instruction tuning backbone, Flat-Linear fusion improves R-Acc. from 78.31% to 86.72% (+8.41%), while TC-S3M fusion improves it from 78.31% to 90.24% (+11.93%) and outperforms Flat-Linear fusion by 3.52%. Therefore, supervised classification improves FGFR, and TC-S3M's further gains support the benefit of modeling food taxonomy semantics.

Appendix A.4.4.3 (Table S10) further reports classifier architectures, computational costs, and classifier-only results. TC-S3M outperforms Flat-Linear in R-Acc. with fewer FLOPs and parameters.

Table 7: Comparison of TC-S3M with other supervised classifiers on Relaxed Accuracy (%).

\hline \text{Model}&\text{No fusion}&\text{Flat-Linear}&\text{Flat-MLP}&\text{Hier-MLP}&\text{TC-S3M}\\\\ \hline \text{Classifier only}&--&71.46&70.76&71.30&\mathbf{74.37}\\\\ \hline \text{Baseline}&57.21&64.87&64.09&64.98&\mathbf{66.89}\\\\ \text{Baseline + I.T.}&70.75&77.90&76.39&77.93&\mathbf{81.54}\\\\ \text{CIPSR + I.T.}&78.31&86.72&85.35&86.94&\mathbf{90.24}\\\\ \hline \end{array}$$ **2. TC-S3M with instruction-tuned Qwen3-VL and without MLLM fusion** In revised Table 5 (Section 4.3), TC-S3M without MLLM fusion achieves 74.37% in R-Acc. Fusing TC-S3M with instruction-tuned Qwen3-VL improves R-Acc. to 81.54% (+10.79%). With TC-S3M fusion, adding CIPSR improves R-Acc. from 81.54% to 90.24% (+8.70%). > Table 5: Effects of CIPSR and TC-S3M (%). $$\begin{array}{lrrr} \hline \text{Configuration}&\text{R-Acc.}&\text{Rot-Acc.}&\text{Rot-Hit}\\\\ \hline \text{TC-S3M only}&74.37&74.37&74.37\\\\ \hline \text{Baseline}&57.21&49.40&57.11\\\\ \quad \text{+ TC-S3M}&66.89&64.24&66.75\\\\ \hline \text{I.T.-only}&70.75&70.06&70.82\\\\ \quad \text{+ TC-S3M}&81.54&81.02&81.78\\\\ \hline \text{CIPSR + I.T.}&78.31&76.65&78.32\\\\ \quad \text{+ TC-S3M (Ours)}&\mathbf{90.24}&\mathbf{89.72}&\mathbf{90.23}\\\\ \hline \end{array}$$ **3. Annotation models, category information, and generation budgets of attribute generation methods** All attribute generation methods use GPT-5.6 Luna (medium) with three-level taxonomy priors: L1 food clusters, L2 ingredient families, and L3 processing form (Section 3.2 and Appendix A.2). Fine-grained category names are excluded. Each method uses Qwen3-VL as the recognition backbone with two-stage fine-tuning. CIPSR additionally uses class representative images as Glancing priors. Revised Table 4 (Section 4.3) reports each method's priors, generation budgets (API calls, tokens, and costs), and MCQ results. Furthermore, to assess gains beyond computational cost, image priors, and self-verification, we add Self-Refine (Madaan et al., 2023) to FineDefics, using the same class representative images during attribute discovery and three-stage verification of visual support, coverage, and consistency at comparable cost. CIPSR outperforms Direct and FineDefics in R-Acc. by 7.75% and 7.42%, respectively. CIPSR also outperforms FineDefics with Self-Refine by 4.29% in R-Acc. at a similar cost (USD 4.34 vs. USD 4.24 per 1,000 images). These results demonstrate the effectiveness of CIPSR's structured glancing–focusing–reasoning for constructing discriminative food attributes under the same annotation model and priors and a comparable generation budget. Appendix A.4.3 (Tables S5–S6) reports the API token usage and cost of the complete CIPSR process by stage and agent. Attribute generation costs USD 728.47 in total. > Table 4: MCQ results (%) and computational cost of attribute generation methods. $$\begin{array}{llrrrrrr} \hline \text{Method}&\text{Prior}&\text{Calls/Img.}&\text{Tokens/Img.}&\text{Cost/1K (USD)}&\text{R-Acc.}&\text{Rot-Acc.}&\text{Rot-Hit}\\\\ \hline \text{Baseline + I.T.}&--&0&0&0&70.75&70.06&70.82\\\\ \text{Direct}&\text{Taxonomy}&1.00&\text{2,013}&0.46&70.56&69.68&70.62\\\\ \text{FineDefics}&\text{Taxonomy}&2.01&\text{3,592}&1.26&70.89&70.12&70.93\\\\ \hline \text{FineDefics + Self-Refine}&\text{Taxonomy + class representative images}&5.30&\text{10,872}&4.24&74.02&73.85&74.13\\\\ \hline \text{CIPSR (Ours)}&\text{Taxonomy + class representative images}&5.57&\text{11,205}&4.34&\mathbf{78.31}&\mathbf{76.65}&\mathbf{78.32}\\\\ \hline \end{array}$$

Comments on the paper

1 comment

We sincerely thank all reviewers and chairs for their time and constructive feedback on Glancing, Focusing, Reasoning: A Cognition-Inspired Framework for Fine-Grained Food Recognition. We particularly appreciate the reviewers' recognition of our work's strengths and their suggestions for additional evaluation evidence.

Overall, the reviewers have provided positive comments on the motivation of our work, the complementary modeling of food attributes and taxonomy semantics, the effectiveness of CIPSR and its generalization to held-out categories, and the clear writing and helpful figures. For the concerns, the reviewers mainly requested additional controlled comparisons, broader evaluations, analysis of ancestor error propagation, and more details on annotation settings, computational costs, and human review.

In conclusion, additional evaluations, cost analyses and annotation and review details have been added during the response stage.

Major Additions

  1. We added supervised classifier comparisons with Flat-Linear, Flat-MLP and Hier-MLP to assess taxonomy modeling beyond classification supervision in Section 4.3 (Table 7) and Appendix A.4.4.3 (Table S10).

  2. We also included full-label-space recognition, using closed-set linear probing and unconstrained category prediction in Section 4.3 (Table 6), MCQs with 16 and 64 candidates in Appendix A.4.1 (Table S2), and cross-dataset generalization on DiningBench, CNFOOD-241 and FoodX-251 in Appendix A.4.7 (Table S16).

  3. We also included annotation cost and budget analyses, reporting API usage and costs by stage and agent and recognition under different annotation budgets in Appendix A.4.3 (Tables S5-S7).

  4. We also included training-time and inference-time ancestor-conditioning ablations to assess the training strategy and error propagation in Appendix A.4.4.4 (Tables S11-S12).

  5. We also included sensitivity analyses of the fusion weight and taxonomy loss weights in Appendix A.4.5 (Tables S13-S14).

Major Revisions

  1. We clarified module contributions: Table 5 in Section 4.3 now includes TC-S3M without MLLM fusion and TC-S3M fusion with instruction-tuned Qwen3-VL.

  2. We clarified baseline fairness: Sections 4.1 and 4.3 specify annotation models, priors and generation budgets. Table 4 (original Table 3) now includes FineDefics with Self-Refine using the same class representative images as CIPSR at comparable cost.

  3. We clarified ancestor conditioning: Section 3.2 specifies the switch from ground-truth to predicted ancestors in the final training epoch to address the training-inference mismatch.

  4. We also included closed-source model comparisons with GPT-5.6 SOL, Gemini 3.1 and Claude Sonnet 4.6 in Table 1 and updated the corresponding analysis in Section 4.2.

  5. Finally, we expanded the human annotation and review details in Appendix A.1, accompanying the existing Table S1, and Appendix A.4.2 to explicitly describe review procedures, statistics and gold-standard annotations.

Table renumbering. The original zero-shot Table 4 is now Table 3. Original supplementary Tables S2, S3, S4, S5 and S6 are now Tables S3, S4, S8, S9 and S15, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.