acceptodds
Withdrawn from ICLR 2027

LatentNutri: Anchor-Driven Latent Computation for Efficient Food Nutrition Estimation in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image-based food nutrition estimation requires open-domain visual understanding and reliable continuous prediction. Existing systems typically rely on CNN/ViT regressors, which often generalize poorly beyond their training distribution, or on vision-language models (VLMs), which predict nutrition values through slow autoregressive text decoding. We introduce LatentNutri, an Anchor-Driven Latent Computation framework for VLM-based food nutrition estimation. LatentNutri uses Single-Pass Latent Forward to compute target-specific anchor states in one VLM forward pass, and Head-Aware Latent Regression to map these states to continuous nutrition values with lightweight regression heads. This design predicts a structured nutrition vector without autoregressive decoding. Training combines an autoregressive SFT warm-up with Scale-aware Physics-guided Regression, which jointly optimizes scale-normalized numerical accuracy and physical plausibility. Experiments on Nutrition5K and DiningBench show that LatentNutri improves prediction quality over CNN/ViT and autoregressive VLM baselines while achieving a 14.9× inference speedup over autoregressive decoding. Code is provided in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Latent Computation, Continuous Regression, Food Nutrition Estimation, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EFhFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentNutri which is a VLM-based food nutrition estimation method. The architecture consists of pretrained VLM (Qwen3-VL) and additional heads to compute nutirition values directly. The training steps consists of two steps: prompt-based VLM fine-tuning and regression-based nutirition estimation head training. This two step training seems to be novel of this paper. By the experiments, LatentNutri outperformed the non-finetuned/SFT-fine-tuned exising open models and commercial models.

Critical strengths
  • This paper proposed the strong nutrition estimation model. By the experiments, it outperformed the baselines.
  • The authors created (and will release) the new nurtition-annotated food dataset,
Critical weaknesses
  • No comparison to the state-of-the-art. In fact, OmniFood8K (which was presented at CVPR2026 (although this paper referred to it as an arXiv paper. ) achived 14.1 % PMAE at calorie estiation of Nutrition 5k. Compared with this, the result by LatentNutri was much lower. As shown in the table in OmniFood8k paper, many methods can be used as baselines. However, in LatentNutri paper, only non-fine-tuned/SFT-fine-tuned OSS and comercial models were used as baselines.

  • Single evaluation is not a novel idea. CNN/ViT baselines are also single evaluation models. The proposed architecture and two-step training method is not novel. Some works adopted RL-based finetuning after SFT-fine-tuning. RL-based training should be incorporated as well.

Questions for authors
  • Please compare the results of LatentNutri with the results shown in Table 2 of the OmniFood8k paper.

  • What if RL-based training is incorpated ?

Reviewer_4w1jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentNutri for image-based food nutrition estimation. The framework consists of Single-Pass Latent Forward and Head-Aware Latent Regression. The former processes predefined nutrition anchors with the image in one VLM pass; the latter maps the resulting states to continuous values through separate regression heads. Extensive experiments on Nutrition5K and DiningBench demonstrate the effectiveness of the proposed method. Inference-time experiments show a substantial speed advantage, with a 14.9-fold speedup over the autoregressive baseline.

Critical strengths

1.The method addresses two intrinsic difficulties in nutrition estimation: large differences in the scales of the targets and constraints linking energy, mass, and macronutrients. Incorporating both into the learning objective connects predictive accuracy with nutritional consistency, an important consideration when learning quantitative properties from visual evidence. 2.Experiments on Nutrition5K and DiningBench demonstrate the method's effectiveness across different food-image distributions. The analysis of nutritional consistency also supports its ability to produce estimates that satisfy known nutritional relationships.

Critical weaknesses

1.Insufficient methodological novelty. The central idea largely follows FoodLMM, which already predicts continuous nutrition values from nutrition-token hidden states using an MLP and regression heads. The proposed method makes only incremental changes to this approach and does not offer a sufficiently substantive technical contribution.

2.Unconvincing experimental comparisons. First, the models are not trained for the same number of epochs: LatentNutri receives one epoch of SFT and three epochs of regression training, while the SFT baselines receive three epochs (Section 4.1). Second, LatentNutri uses SNAE and physical regularization, whereas the CNN/ViT baselines use MSE (Appendix A.2). Removing SNAE raises DiningBench PMAE from 26.68% to 32.89% (Table 3), showing that performance is strongly affected by the loss function. Without comparisons under consistent training conditions, the experiments do not convincingly demonstrate an advantage of the proposed regression design.

Questions for authors
  1. Please clarify the novelty of the proposed method and its fundamental distinctions in design rationale from FoodLMM and other related methods.

  2. Are results available comparing LatentNutri with the SFT baselines after the same total number of training epochs, and comparing the regression models using the same loss function? Do the gains in Tables 1 and 2 persist in these comparisons?

  3. In Tables 6 and 7, how can the effect of field semantics be distinguished from the effects of anchor position and template structure?

  4. How was the numerical accuracy of NutriFood-75K's nutrition labels assessed, beyond the review of 100 samples for obvious inconsistencies described in Appendix C.1?

  5. Under Equation (21), the calorie results in Table 2 imply mean ground-truth values of 270.5 kcal for Google-Nutrition-rgb and 254.6 kcal for LatentNutri. Were these results computed on the same samples with the same averaging procedure, and what accounts for this discrepancy?

Reviewer_QLcCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

See below

Critical strengths

S1. Significant improvements in inference efficiency while maintaining competitive predictive performance. LatentNutri replaces autoregressive nutritional value generation with single-pass latent regression, achieving approximately 14.9× faster inference while maintaining competitive prediction performance.

S2. Systematic experimental validation and substantial dataset construction efforts. The paper presents a relatively comprehensive set of experiments and ablation studies on Nutrition5K and DiningBench. It also introduces NutriFood-75K, demonstrating considerable effort in dataset construction and a reasonably thorough experimental evaluation.

Critical weaknesses

W1. The methodological novelty of Anchor-Driven Latent Computation beyond existing task-specific token regression methods remains insufficiently established. FoodLMM already employs nutrition-specific token hidden states, followed by an MLP and corresponding regression heads, to predict continuous nutritional values. The primary additional design in LatentNutri is to predefine target-specific anchors within a semantic template and directly extract their hidden states through Single-Pass Latent Forward. Although the paper separately demonstrates the effectiveness of the semantic anchor template and regression head design, the current experiments do not sufficiently disentangle the individual contributions of semantic anchors, target-specific independent regression heads, and prefilled single-pass computation. Since Anchor-Driven Latent Computation constitutes the paper's core methodological contribution, this lack of evidence makes it difficult to assess its novelty and independent methodological value.

W2. The reliability of nutritional annotations in NutriFood-75K lacks sufficiently rigorous independent quantitative validation. The nutritional values in NutriFood-75K are primarily estimated automatically by a vision-language model (VLM) and filtered using criteria such as confidence scores and physical plausibility. The manual inspection described in Appendix C.1 covers only 100 randomly sampled image–annotation pairs, with reviewers relying mainly on general nutritional knowledge to identify obvious anomalies. No quantitative comparison against independent, reliable nutritional ground truth is provided. Consequently, the current results demonstrate more convincingly that the annotations are useful for model training than that the underlying nutritional values are quantitatively accurate and reliable.

Questions for authors

Q1. Could the authors conduct a factorial ablation study that independently evaluates the effects of semantic anchor prompts and target-specific independent regression heads, while keeping the backbone, training data, loss functions, and training budget fixed? For example, a four-way comparison covering all combinations of these two design choices would help isolate the individual contributions of anchor prompts and independent regression heads, as well as their interaction effects.

Q2. Compared with the nutrition-specific token hidden-state regression already employed by FoodLMM, could the authors provide controlled experiments or more direct evidence to establish the independent methodological contribution of Anchor-Driven Latent Computation? In particular, beyond eliminating autoregressive decoding and improving inference efficiency, do predefined anchors and Single-Pass Latent Forward offer measurable advantages in prediction accuracy or representation quality?

Q3. Could the authors construct an independently validated subset of NutriFood-75K with reliable nutritional reference values and quantitatively report the prediction errors and systematic biases for calories, food mass, fat, carbohydrates, and protein? Such an evaluation would provide stronger evidence for the numerical reliability of the proposed dataset and clarify its suitability for nutritional estimation research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.