acceptodds
Under review as a conference paper at ICLR 2027

Triage Before Fine-Tuning: Feasibility-Gated Selection and Monitored Adaptation of Embodied Foundation Models for a Fixed Physical Platform

Abstract

Embodied foundation models are increasingly deployed by selection: a practitioner picks a public checkpoint by benchmark rank and fine-tunes it on local data. The platform, however, is fixed before the model is chosen: its sensors, onboard compute and operating site are set by the task, and at the start there is little or no local data. We ask what such a checkpoint loses on a fixed physical platform and which part of the loss local data can buy back. Our answer is to split the gap by what can repair it. A sensing gap, information the sensors do not capture, cannot be closed with their data; a latency gap, output arriving after the control loop needs it, belongs to the loop, not the weights; only a distribution gap is fine-tuning's to repair. We turn this into TRIAGE, a procedure for feasibility-gated selection and monitored adaptation in five stages: gates on the sensor contract and on latency that no score can override, ten-minute local probes that place candidates in tiers, a target-relevant pick within the top tier, and fine-tuning monitored for tracking error, perturbation sensitivity and reliance on the image. In Bench2Drive, a 500 ms deadline below its latency makes the closed-loop champion collide on 80 to 90% of routes; on NAVSIM real logs, losing field of view moves plans 2.2 to 5.7 times more than losing resolution at equal pixels (1.6 to 2.4 on Bridge V2 manipulation), 20k local frames close 75% of a stability gap but raise perturbation sensitivity up to 3.4 times, and picking by score or by champion violates a gate in two and four of four deployment profiles. In a field test on a full-scale vehicle, the factory camera failed the sensing gate, the prescribed added camera repaired the view, laboratory latencies did not carry over on board, and in closed loop plans rested on images over two seconds old, the one stale-command takeover coming under command hold. More broadly, it lets organizations that cannot train foundation models from scratch find out, before buying data or hardware, whether a public checkpoint can run on the robot or vehicle they own, and what to fix if not.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied foundation models, vision-language-action models, model selection, feasibility gating, fine-tuning, domain adaptation, sensor constraints, inference latency, autonomous driving, robot manipulation, real-world field study

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m3HmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem of selecting pre-trained foundation models for a fixed physical deployment platform, with fixed sensors, on-board compute, and task specification. The authors categorize gaps into three types: sensing gap, latency gap, and distribution gap. Based on the analysis, the conclusion reveals that only the distribution gap can be mitigated by fine-tuning. Based on this taxonomy, the authors propose the TRIAGE procedure, which conducts probing before full fine-tuning to select candidate models. The authors demonstrate this framework with multiple open-source model benchmarks and autonomous driving experiments.

Critical strengths
  1. The problem statement is practically meaningful. The paper addresses a critical engineering question: given a fixed deployment platform, task definition, sensor configuration, and compute budget, how practitioners should select open-source models as initial baselines. The authors decompose the bottleneck into three orthogonal dimensions: sensor limitation, inference latency, and distribution shift, which provides a clean conceptual framework for model selection.

  2. The authors conduct systematic empirical evaluations by reproducing and benchmarking multiple public open-source models across the three proposed dimensions, which supports their core argument with empirical evidence.

Critical weaknesses
  • The practical value of this work for industrial mass-production scenarios is not sufficiently justified, especially for autonomous driving. For production teams, internal datasets are much larger than public datasets. In practice, companies often reuse only the model architecture from open-source research, rather than the pretrained weights. When training with massive in-house data, the contribution of initial public weights and original pretraining settings becomes unclear. The paper does not discuss whether the TRIAGE workflow still brings tangible benefits under this common industrial setup.

Overall, this work reads more like a position paper that articulates a framework for model selection under hardware constraints. My concern mainly lies in the practical value of this work, especially given that industrial mass-production scenarios typically have much more data than the publicly available models.

Questions for authors
  • For industrial autonomous driving teams with massive proprietary datasets, practitioners usually adopt open-source model structures but discard public pretrained weights and train from scratch / with in-house pretraining. Under this setting, does your TRIAGE pipeline still deliver meaningful gains? Could you elaborate on how much the initial public weights matter when fine-tuning on large-scale internal data?

  • The paper suggests the possibility of choosing hardware based on public model performance. Could you clarify the scope of your claim? Do you intend that your method is only for model selection on already-fixed hardware, or can it genuinely guide hardware specification decisions for mass-production products?

  • What is the core actionable outcome of TRIAGE for production-level autonomous driving pipelines? Could you quantify the practical benefits (e.g., saved engineering time, better final performance) compared with naive baseline selection?

Reviewer_p7mSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TRIAGE, a framework for selecting and adapting embodied foundation models to a fixed physical platform. The main idea is to distinguish sensing, latency, and distribution gaps before deciding whether fine-tuning is appropriate. The authors evaluate the framework on multiple driving and manipulation models, including a real-vehicle study.

Critical strengths

I find the problem relevant and appreciate the broad experimental effort. The results provide several useful observations about model deployment, particularly the importance of sensor compatibility and inference latency.

Critical weaknesses

My main concern is the novelty of the proposed framework. Checking sensor compatibility and inference latency before selecting a model seems like a reasonable engineering practice. While organizing these checks into a systematic workflow is useful, I am not sure this is sufficient as a methodological contribution. It would be helpful to explain more clearly what TRIAGE provides beyond a conventional deployment checklist.

The model selection component is not particularly convincing. In Section 4.4, the proposed probes actually perform worse than selecting the highest-scoring model within the top tier, with a mean regret of 0.48 compared with 0.18. As a result, most of the benefit appears to come from excluding incompatible models rather than improving model selection. I would like to see a comparison against a simpler baseline that checks sensor and latency requirements before applying benchmark-based ranking.

I find the distinction between the three types of gaps interesting, but the separation seems stronger in the theoretical formulation than in practice. For example, fine-tuning can compensate for much of the degradation caused by moderate image cropping, while adaptation can also change inference latency. These factors may interact in real systems. The paper should better explain how the dominant gap is identified when several factors contribute to the same failure.

Some experimental conclusions rely mainly on changes in model outputs rather than actual task performance. For example, the comparison between field-of-view reduction and resolution degradation measures how much the predicted trajectory or action changes. However, a larger change does not necessarily mean worse performance. Although the paper includes some closed-loop tests, I would like to see a stronger connection between these sensitivity measurements and actual safety or task success.

I also have concerns about the latency gate. The framework uses median inference latency, but the real-vehicle experiment shows that passing this check does not necessarily mean the overall system meets its deadline. For DriveVA, the measured inference latency was below the 1.2 s threshold, yet plans arrived approximately every 1.5 s during closed-loop operation. This suggests that end-to-end latency and its variability may be more important than the model's inference time alone.

The real-world experiment is valuable, but its evidence is still limited. The framework ultimately selected no model for deployment, and the fine-tuning stage was not evaluated on the vehicle. The subsequent closed-loop study used only two runs per controller at low speed. Therefore, the field study mainly demonstrates the detection of deployment problems, rather than successful model selection and adaptation through the complete workflow.

Questions for authors

How much does TRIAGE improve model selection compared with a simpler procedure that first checks sensor and latency compatibility and then selects the model with the best benchmark score? I think this would help clarify the contribution of the proposed probes.

Why is median inference latency used for the feasibility gate? Given the real-vehicle results, would measuring end-to-end latency, including preprocessing, communication, and control execution, provide a more reliable criterion?

How do the authors distinguish between an irrecoverable sensing gap and a performance gap that can be compensated through fine-tuning? The TCP results suggest that some fairly substantial sensing degradation can be largely recovered.

Can the authors provide more direct evidence that the proposed perturbation sensitivity measures predict actual closed-loop failures? In particular, does a model with a larger trajectory displacement under image cropping consistently suffer a greater reduction in driving performance?

Since the real-vehicle evaluation did not result in a selected and adapted model, could the authors clarify which parts of the complete TRIAGE workflow have been validated in a successful deployment setting?

1 comment

Thanks for the detailed response and for running the additional experiments. I went through the new results, and I think they address quite a few of my earlier concerns. In particular, the end-to-end latency measurements and the additional closed-loop tests are helpful. I also appreciate that you clarified the role of the probes and acknowledged that they do not outperform the simpler gated ranking baseline.

I still have some concerns about the novelty though. From the revised results, it seems that most of the improvement comes from ruling out models that do not meet the sensing or latency requirements, and then using the benchmark scores for selection. I agree this is useful in practice, but I am still not fully convinced that it represents a sufficiently new methodology.

The real-world experiments are a good addition, although the evaluation is still on a fairly short route. Also, the fine-tuned model actually failed in closed loop, so the adaptation part of the framework is not yet very convincing to me.

One other small point is the latency threshold. ReCogDrive exceeds the 1.2 s limit by only 4 ms. I am not sure this difference is meaningful enough to explain its failure on the vehicle, and I would be careful about drawing a strong conclusion from that comparison.

Overall, I think the paper is stronger than the original submission, and I appreciate the effort put into the rebuttal. My main reservation is still about how much of the contribution is a new method rather than a systematic evaluation of existing deployment practices.

Reviewer_8VLROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The goal of this paper is to aid in understanding what foundation model checkpoints should be selected to apply to a new robot platform in the physical domain. The approach is to look to gaps in sensing, or latency, to determine whether the gap can be closed by adapting the foundation model to the new platform. Given that many foundation models are tested or developed with sensing and computing or dynamical hardware that will differ from the selected deployment platform, there can be uncertainty into how to adapt (or even select) what foundation model for the physical platform to use in execution. The paper describes how the approach learns that the existing camera sensor is unable to be used, thus a new sensor is prescribed, and this improves the behavior of the system.

  • The selected research question is a great fit: if I cannot train my own foundation model, where should I start? Further, can I determine a priori that a certain model will be certain not to work, and thus save time by not exploring it? The results rely on a gated selection process, where gate failure reports in sensing or latency are translated into potential "fixes" (repairs, as described in the paper). These repairs may be an additional sensor, or a model that can improve the existing sensor's behavior.
  • The simulated results with CARLA are translated to a closed-track test that was able to validate the work. The closed-track work demonstrates success of the work.
Critical strengths
  • The selected research question is a great fit: if I cannot train my own foundation model, where should I start? Further, can I determine a priori that a certain model will be certain not to work, and thus save time by not exploring it? The results rely on a gated selection process, where gate failure reports in sensing or latency are translated into potential "fixes" (repairs, as described in the paper). These repairs may be an additional sensor, or a model that can improve the existing sensor's behavior.
  • The simulated results with CARLA are translated to a closed-track test that was able to validate the work. The closed-track work demonstrates success of the work.
Critical weaknesses
  • There are a few citations to a paper in 1953, including some inferences that this paper can provide insights into policies. This is a bit of a stretch, since RL policies were not available at the time of that paper; to make such a claim ("By Blackwell’s comparison of experiments (Blackwell, 1953), no policy computed from a coarser channel, including one fine-tuned through it, can reach a lower risk than the best policy on the richer channel, and it stays strictly above whenever no optimal decision can do without what was removed (Prop. 1, Appendix B.2)"). Such claims should be attributed to the proofs in the paper or appendix which extend the original result to apply to RL policies, and not a paper's results from decades before RL policies were first defined. Otherwise, they should more clearly indicate whether the limits in the channels are what matters, not the application of RL or some other technology on top of that channel.
Questions for authors
  • Why is it that a coarser channel is guaranteed to not work, compared to a richer channel, if the latency of the richer channel is much much larger (or if there is a model to recover information from the coarser channel)? With recent approaches to recover or predict details from coarse channels where side-channel information may be of use, then it is not clear why the richer channels are the only way to succeed. For example, a 4k resolution video feed at 120fps may require so much time to compute that it cannot return answers in time---while a radar sensor at 20Hz can provide both distance and relative velocity of a lead vehicle's behavior, due to the inherent benefits of the radar calculations. It seems that the channel to be described may not be well understood by me, but the information gained is the key piece--not the equipment or data stream.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.