- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents GFRFood, a cognition-inspired framework for fine-grained food recognition using multimodal large language models (MLLMs). The proposed GFRFood framework first introduces Cognition-Inspired Progressive Semantic Reasoning (CIPSR) by employing multiple LLM-based agents to generate discriminative food attributes through three progressive stages: glancing, focusing, and reasoning. In addition, it further includes Taxonomy-Conditioned Semantic State-Space Model (TC-S3M) models hierarchical food categories through selective state transitions, providing complementary category predictions that are fused with the MLLM outputs. The authors also construct FoodFG, a benchmark curated from several existing datasets using MCQ. Experiments show that GFRFood outperforms general-purpose and food-specific MLLMs on MCQ for fine-grained food recognition.
Critical strengths
- The three challenges identified in the introduction are reasonably motivated, and the examples in Figure 1 help illustrate why conventional visual-semantic understanding can be insufficient.
- The combination of ingredient recognition, visual attribute generation, and hierarchical category modeling is intuitively meaningful for this task.
- The paper writing is easy to follow and the figures are illustrative and helpful.
Critical weaknesses
- A key concern is that the final performance improvement is not solely attributable to the proposed progressive reasoning mechanism. According to Table 5, CIPSR achieves 78.31% R-acc, while combining it with TC-S3M increases accuracy to 90.24%. TC-S3M itself is a separately trained supervised classifier operating on visual features and hierarchical category labels.
- The proposed method is much more complex compared with existing methods, combining several components, such as attribute-based fine-grained recognition, multi-agent verification, hierarchical classification and selective state-space models. These components are not new in general, though the proposed method composes these into a framework.
- The main experiments use four-way MCQ on FoodFG, with training and test images drawn from the same set of food categories. While this is a reasonable evaluation protocol, it does not fully establish generalization to more realistic recognition scenarios involving unseen dishes, varying candidate sets, or unconstrained category prediction. It would be good to see more generalization experiments except for the zero-shot evaluation in Table 4.
- Adding some results of the latest closed-source models for reference would be helpful, such as GPT, Gemini and Claude. Please list the computational cost comparison with the existing methods as well.
- It would be great to add more details about human annotations and review, as well as more stats for this process.
Questions for authors
Please see the weaknesses.
2 comments
Part 2. Generalization, closed-source models, costs, and human review
3. Generalization across datasets, varying candidate sets, and unconstrained category prediction
We extend our evaluation with cross-dataset experiments, MCQs with larger candidate sets, and unconstrained generation.
1) Cross-dataset generalization. In Appendix A.4.7 (Table S16), we evaluate the cross-dataset generalization of our method on DiningBench-4, DiningBench-8, CNFOOD-241, and FoodX-251 without target-dataset fine-tuning. Compared with Qwen3-VL, CIPSR achieves R-Acc. of 67.57% (+7.24%), 65.93% (+7.30%), 76.48% (+5.17%), and 80.75% (+6.75%), respectively.
\hline \text{Model}&\text{DiningBench-4}&\text{DiningBench-8}&\text{CNFOOD-241}&\text{FoodX-251}\\\\ \hline \text{Qwen3-VL}&60.33&58.63&71.31&74.00\\\\ \text{Qwen3-VL + I.T.}&55.93&51.04&71.93&68.75\\\\ \text{CIPSR}&\mathbf{67.57}&\mathbf{65.93}&76.48&\mathbf{80.75}\\\\ \text{CIPSR + I.T.}&61.21&59.76&\mathbf{76.91}&78.60\\\\ \hline \end{array}$$ **2) Larger MCQ candidate sets.** At candidate set sizes of 16 and 64 (Appendix A.4.1, Table S2), GFRFood outperforms the instruction-tuned baseline by 18.23% and 18.50% in R-Acc., respectively. Its advantage holds with larger candidate sets. > Table S2: Relaxed accuracy (%) on text-based MCQs across candidate set sizes. $$\begin{array}{lrrr} \hline \text{Model}&K=4&K=16&K=64\\\\ \hline \text{Qwen3-VL}&57.21&42.15&31.55\\\\ \text{Qwen3-VL + I.T.}&70.75&69.93&67.61\\\\ \text{CIPSR + I.T.}&78.31&76.61&75.02\\\\ \text{GFRFood}&\mathbf{90.24}&\mathbf{88.16}&\mathbf{86.11}\\\\ \hline \end{array}$$ **3) Unconstrained category prediction.** Revised Table 6 (Section 4.3) evaluates all 1,622 categories. The MLLM generates a category name using only the image and a fixed question, without a category list or TC-S3M fusion. We match the category name in the first 20 generated tokens to the ground truth using semantic similarity. CIPSR with instruction tuning improves Top-1 accuracy from 65.61% to 77.91% (+12.30%) over the instruction-tuned baseline. > Table 6: Top-1 accuracy (%) for unconstrained category prediction. $$\begin{array}{lr} \hline \text{Model}&\text{Top-1 Acc.}\\\\ \hline \text{Baseline}&45.84\\\\ \text{Baseline + I.T.}&65.61\\\\ \text{CIPSR}&66.85\\\\ \text{CIPSR + I.T.}&\mathbf{77.91}\\\\ \hline \end{array}$$ **4. Closed-source model results and computational cost** Revised Table 1 (Section 4.2) adds three closed-source models. GFRFood surpasses GPT-5.6 SOL by 20.86% and 21.01% in R-Acc. on text-based and prototype-based MCQs, respectively. > Table 1: Relaxed accuracy (%) of GFRFood and closed-source MLLMs on FoodFG. $$\begin{array}{lrr} \hline \text{Model}&\text{Text-based MCQs}&\text{Prototype-based MCQs}\\\\ \hline \text{GPT-5.6 SOL}&69.38&71.17\\\\ \text{Gemini 3.1}&67.85&70.06\\\\ \text{Claude Sonnet 4.6}&62.27&64.54\\\\ \text{GFRFood}&\mathbf{90.24}&\mathbf{92.18}\\\\ \hline \end{array}$$ We report costs for offline attribute annotation and online inference. **1) Offline attribute annotation.** Revised Table 4 (Section 4.3) compares the annotation costs of CIPSR and existing attribute generation methods. CIPSR’s offline annotation costs USD 728.47 (USD 4.34 per 1,000 images). At comparable cost, CIPSR outperforms FineDefics with Self-Refine by 4.29% in R-Acc. > Table 4: Annotation costs and relaxed accuracy (%) of attribute generation methods. $$\begin{array}{lrrr} \hline \text{Method}&\text{Tokens/Img.}&\text{Cost/1K (USD)}&\text{R-Acc.}\\\\ \hline \text{Direct}&\text{2,013}&0.46&70.56\\\\ \text{FineDefics}&\text{3,592}&1.26&70.89\\\\ \text{FineDefics + Self-Refine}&\text{10,872}&4.24&74.02\\\\ \text{CIPSR (Ours)}&\text{11,205}&4.34&\mathbf{78.31}\\\\ \hline \end{array}$$ **2) Online inference.** For inference-cost comparisons, fine-tuned Qwen3-VL is the backbone reference. CIPSR runs offline, and TC-S3M adds 0.068 GFLOPs per image. **5. Human annotation procedures and review statistics** Appendix A.1 details human review. **1) Category standardization.** Human review removes 457 coarse-grained categories when fine-grained subclasses coexist. **2) Image screening.** Human review removes 1,792 of 2,076 images near the label screening thresholds. **3) Deduplication.** Human review removes 615 of 747 images near the similarity thresholds. **4) Manual review.** One annotator re-examines category definitions and removes 12 categories containing 2,214 images, leaving 284,935 images across 1,622 categories. **5) Final audit.** One annotator checks image quality and image–label consistency in 1,000 randomly sampled images (seed 42). Of these, 947 pass both criteria (94.7%). Appendix A.4.2 reports a gold set of 4,866 images, with three per category. One annotator labels ingredient identities, visible states, and bounding boxes, followed by spot checks.Table S16: R-Acc. (%) on external food benchmarks.
Part 1. Core module contributions and novelty
Rebuttal:
We sincerely thank the reviewer for the constructive suggestions. We address each concern in detail below.
1. Contributions of CIPSR, TC-S3M, and supervised classification
GFRFood's gains come from CIPSR and from both taxonomy semantic modeling and supervised classification in TC-S3M.
1) Contributions of CIPSR and TC-S3M. In revised Table 5 (Section 4.3), TC-S3M without MLLM fusion achieves 74.37% in R-Acc. Fusing TC-S3M with instruction-tuned Qwen3-VL improves R-Acc. to 81.54% (+10.79%). CIPSR with instruction tuning improves R-Acc. from 70.75% to 78.31% (+7.56%) over the instruction-tuned baseline. With TC-S3M fusion, adding CIPSR improves R-Acc. from 81.54% to 90.24% (+8.70%).
\hline \text{Configuration}&\text{R-Acc.}&\text{Rot-Acc.}&\text{Rot-Hit}\\\\ \hline \text{TC-S3M only}&74.37&74.37&74.37\\\\ \hline \text{Baseline}&57.21&49.40&57.11\\\\ \quad \text{+ TC-S3M}&66.89&64.24&66.75\\\\ \hline \text{I.T.-only}&70.75&70.06&70.82\\\\ \quad \text{+ TC-S3M}&81.54&81.02&81.78\\\\ \hline \text{CIPSR + I.T.}&78.31&76.65&78.32\\\\ \quad \text{+ TC-S3M (Ours)}&\mathbf{90.24}&\mathbf{89.72}&\mathbf{90.23}\\\\ \hline \end{array}$$ **2) Contributions of taxonomy modeling and supervised classification in TC-S3M.** Revised Table 7 (Section 4.3) assesses whether TC-S3M benefits from taxonomy modeling beyond classification supervision. Specifically, we compare TC-S3M with three supervised classifiers trained on the same frozen Qwen3-VL visual features, selecting each classifier's best-performing fusion weight $w$. Flat-Linear uses a fully connected layer, Flat-MLP uses residual MLP blocks, and Hier-MLP adds three auxiliary taxonomy heads to Flat-MLP. With the CIPSR and instruction tuning backbone, Flat-Linear fusion improves R-Acc. from 78.31% to 86.72% (+8.41%), while TC-S3M fusion improves it from 78.31% to 90.24% (+11.93%) and outperforms Flat-Linear fusion by 3.52%. Therefore, supervised classification improves FGFR, and TC-S3M's further gains support the benefit of modeling food taxonomy semantics. Appendix A.4.4.3 (Table S10) further reports classifier architectures, computational costs, and classifier-only results. > Table 7: Comparison of TC-S3M with different supervised classifiers on Relaxed Accuracy (%). $$\begin{array}{lrrrrr} \hline \text{Model}&\text{No fusion}&\text{Flat-Linear}&\text{Flat-MLP}&\text{Hier-MLP}&\text{TC-S3M}\\\\ \hline \text{Classifier only}&--&71.46&70.76&71.30&\mathbf{74.37}\\\\ \hline \text{Baseline}&57.21&64.87&64.09&64.98&\mathbf{66.89}\\\\ \text{Baseline + I.T.}&70.75&77.90&76.39&77.93&\mathbf{81.54}\\\\ \text{CIPSR + I.T.}&78.31&86.72&85.35&86.94&\mathbf{90.24}\\\\ \hline \end{array}$$ **2. Novelty analysis** Our main contribution is a set of cognition-inspired mechanisms for MLLMs to capture fine-grained food semantics more accurately and comprehensively. We use attribute-based fine-grained recognition, multi-agent verification, and hierarchical classification to realize three complementary designs. **1) Cognition-inspired visual food understanding.** Inspired by how humans understand food, CIPSR extends attribute-based recognition by decomposing food understanding into Glancing, Focusing, and Reasoning. Glancing establishes global context to discover discriminative cues. Focusing identifies ingredients and their states from local visual evidence. Reasoning integrates visual cues with cooking methods and ingredient relations. This mechanism guides MLLMs to recover missed discriminative cues and disambiguate ingredient identities and states. In revised Table 4 (Section 4.3), CIPSR outperforms the attribute-based method FineDefics with Self-Refine by 4.29% in R-Acc. at comparable annotation cost. **2) Reliable cognitive attribute construction.** Multi-agent orchestration improves attribute reliability by verifying and correcting MLLMs' ingredient recognition and semantic interpretation. Decoupling ingredient identity and state preserves ingredient visual–semantic correspondence under various ingredient forms. The prior-guided and prior-free Extractor branches capture complementary category prior semantics and image-specific visual cues. To mitigate confirmation bias, counterfactual verification assumes each ingredient candidate is incorrect and revises its identity or state if an alternative has stronger local visual support. Appendix A.4.2 (Tables S3–S4) evaluates these agent contributions. **3) Coarse-to-fine food taxonomy modeling.** TC-S3M follows the progression of human food categorization from coarse food clusters through ingredient families to processing form. TC-S3M uses ancestor taxonomy semantics to refine category features from coarse to fine. Appendix A.4.4 (Tables S8–S10) shows that TC-S3M outperforms flat classifiers and conventional hierarchical classification models.