- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the problem of selecting pre-trained foundation models for a fixed physical deployment platform, with fixed sensors, on-board compute, and task specification. The authors categorize gaps into three types: sensing gap, latency gap, and distribution gap. Based on the analysis, the conclusion reveals that only the distribution gap can be mitigated by fine-tuning. Based on this taxonomy, the authors propose the TRIAGE procedure, which conducts probing before full fine-tuning to select candidate models. The authors demonstrate this framework with multiple open-source model benchmarks and autonomous driving experiments.
Critical strengths
-
The problem statement is practically meaningful. The paper addresses a critical engineering question: given a fixed deployment platform, task definition, sensor configuration, and compute budget, how practitioners should select open-source models as initial baselines. The authors decompose the bottleneck into three orthogonal dimensions: sensor limitation, inference latency, and distribution shift, which provides a clean conceptual framework for model selection.
-
The authors conduct systematic empirical evaluations by reproducing and benchmarking multiple public open-source models across the three proposed dimensions, which supports their core argument with empirical evidence.
Critical weaknesses
- The practical value of this work for industrial mass-production scenarios is not sufficiently justified, especially for autonomous driving. For production teams, internal datasets are much larger than public datasets. In practice, companies often reuse only the model architecture from open-source research, rather than the pretrained weights. When training with massive in-house data, the contribution of initial public weights and original pretraining settings becomes unclear. The paper does not discuss whether the TRIAGE workflow still brings tangible benefits under this common industrial setup.
Overall, this work reads more like a position paper that articulates a framework for model selection under hardware constraints. My concern mainly lies in the practical value of this work, especially given that industrial mass-production scenarios typically have much more data than the publicly available models.
Questions for authors
-
For industrial autonomous driving teams with massive proprietary datasets, practitioners usually adopt open-source model structures but discard public pretrained weights and train from scratch / with in-house pretraining. Under this setting, does your TRIAGE pipeline still deliver meaningful gains? Could you elaborate on how much the initial public weights matter when fine-tuning on large-scale internal data?
-
The paper suggests the possibility of choosing hardware based on public model performance. Could you clarify the scope of your claim? Do you intend that your method is only for model selection on already-fixed hardware, or can it genuinely guide hardware specification decisions for mass-production products?
-
What is the core actionable outcome of TRIAGE for production-level autonomous driving pipelines? Could you quantify the practical benefits (e.g., saved engineering time, better final performance) compared with naive baseline selection?
Thanks for the detailed response and for running the additional experiments. I went through the new results, and I think they address quite a few of my earlier concerns. In particular, the end-to-end latency measurements and the additional closed-loop tests are helpful. I also appreciate that you clarified the role of the probes and acknowledged that they do not outperform the simpler gated ranking baseline.
I still have some concerns about the novelty though. From the revised results, it seems that most of the improvement comes from ruling out models that do not meet the sensing or latency requirements, and then using the benchmark scores for selection. I agree this is useful in practice, but I am still not fully convinced that it represents a sufficiently new methodology.
The real-world experiments are a good addition, although the evaluation is still on a fairly short route. Also, the fine-tuned model actually failed in closed loop, so the adaptation part of the framework is not yet very convincing to me.
One other small point is the latency threshold. ReCogDrive exceeds the 1.2 s limit by only 4 ms. I am not sure this difference is meaningful enough to explain its failure on the vehicle, and I would be careful about drawing a strong conclusion from that comparison.
Overall, I think the paper is stronger than the original submission, and I appreciate the effort put into the rebuttal. My main reservation is still about how much of the contribution is a new method rather than a systematic evaluation of existing deployment practices.