acceptodds
Under review as a conference paper at ICLR 2027

Joint Driver, Vehicle, and Road Modeling for Forecasting and Driver Monitoring

Abstract

Predicting how drivers, vehicles, and road scenes interact and evolve together is central to driver monitoring. Prior work models in-cabin activity or traffic-conditioned driver motion in isolation, motivating joint driver, vehicle, and road modeling with real-time on-vehicle evaluation. We introduce TriDrive, to our knowledge the first unified framework that jointly forecasts driver kinematics, vehicle dynamics, and road demands through an automation-conditioned transition model. Modality-specific encoders (an anchored kinematic representation of the driver, causal CAN-bus dynamics, and frozen V-JEPA 2 road latents with structured road margins) are connected by directed residual connections through which driver and road context refine vehicle forecasts. We evaluate TriDrive on three downstream tasks. On the public AIDE benchmark, its kinematic encoder recipe sets a new full-set state of the art (SOTA) among published baselines (48.05 versus 71.47 All-MPJPE). On 197.2 hours of naturalistic BATON driving, directed connections and road margins raise assistance-engaged PR-AUC by 0.084 for steering onset and 0.286 for time-to-collision drops. For real-time use, we distill the road encoders and run TriDrive on a comma four with an external 8 GB GPU, where a lightweight current-state warning probe updates at 5 Hz with 177 ms p95 latency while the joint model forecasts concurrently. The probe is above an openpilot-based baseline on human-labeled manual-driving warnings (AUROC 0.725 versus 0.563), and in a paired on-road study 14 drivers rate its warnings as more appropriate (+1.79) and timely (+2.67) than those of openpilot's driver-monitoring system.

Primary area:
applications to robotics, autonomy, planning
Keywords:
joint forecasting, driver monitoring, human-centered driving, multimodal representation learning, on-vehicle deployment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bdt1OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents TriDrive, a framework for forecasting driver behavior, vehicle dynamics, and road demands from synchronized cabin observations, vehicle signals, and road context. Its main design connects separately pretrained branches through directed residual connections, conditioned on the automation state. The evaluation covers pose forecasting on AIDE, event forecasting on BATON, and an additional set of unseen drivers. The authors also deploy the system on a vehicle and conduct a paired study with 14 participants. The deployed warning classifier uses current-state features, while the joint forecasting model runs alongside it.

Critical strengths

The forecasting evaluation is fairly thorough. The driver-disjoint BATON split and the frozen-model evaluation on 60 additional drivers and 42 non-overlapping vehicles are particularly useful. These tests provide reasonable evidence that the gains on selected forecasting targets extend beyond the original test split.

The deployment work adds practical value. The authors report measured on-vehicle performance, including a 5 Hz warning readout with 177 ms p95 latency while forecasting runs concurrently. They also report results that do not favor the method, including the lack of consistent warning gains from forecast features and the increased annoyance in the user study. This makes the contribution and its limits easier to assess.

Critical weaknesses

My main reservation is the evidence for the joint architecture itself. The AIDE result comes from a separate pose forecaster, and the deployed warning classifier does not use the joint model’s forecasts. Adding forecast features does not consistently improve warning AUROC (Appendix D.2). The BATON results do support useful forecasting gains, but all-to-all fusion is competitive and sometimes better in Table 3. The paper would be stronger if it stated more clearly which benefits require the proposed directed connections, and which come from the input representation or training recipe. At present, I find the empirical contribution stronger than the architectural novelty.

The warning results are encouraging, but the practical advantage remains qualified. The offline openpilot-based baseline learns to reproduce logged alerts rather than replaying the actual policy, which complicates the comparison. The advantage also reverses under assistance-engaged driving: AUROC is 0.664 versus 0.746 for the baseline. In the on-road study, appropriateness and timeliness improve, but annoyance increases and overall preference remains inconclusive. Per-participant driving exposure and warning rates would help interpret these results, especially given the different aggregate recording durations reported in Appendix H

Questions for authors

Do you have an existing comparison between directed connections and an equally sized fusion baseline using the same inputs and training schedule? This would help clarify the benefit of the connection design beyond its smaller parameter count relative to the reported all-to-all model.

How closely does the offline openpilot-based surrogate reproduce actual logged alerts on held-out drivers? If this has been measured, please report it so that the warning comparison can be interpreted more directly.

From the existing study logs, could you provide the paired driving durations and warning rates for each participant? Please also clarify how the 5.9 versus 2.5 hours of device recordings relate to the drives included in the paired analysis, and whether alert sound and display settings were matched.

2 comments

Thank you for the constructive assessment. We have added matched fusion controls, measured surrogate fidelity, and clarified the on-road study.

1. Comparing directed connections with equally sized fusion.

We added a concatenation control combining driver and road states with the vehicle latent, using 575,616 versus 576,768 connection parameters for directed fusion, a 0.2% difference. Inputs, pretrained branches, training schedule, model selection, and minibatch sequence are matched across three seeds.(Appendix B.6, Table 10).

The control improves PR-AUC over independent branches by 0.037/0.028/0.042 for manual steering/gas/brake and 0.079 for engaged steering, versus 0.030/0.020/0.037 and 0.084 for directed fusion. Concatenation is higher by 0.007 ± 0.002 on manual steering; other designated benchmark differences do not meet our criterion. On 60 additional drivers, its steering gains are 0.049/0.072 over independent branches, with neither steering difference from directed fusion meeting the criterion.

Simpler fusion can therefore integrate this information effectively. We retain directed connections because driver actions and road demands inform vehicle motion, and explicit paths preserve separate driver/road predictions with 29.8% fewer connection parameters than all-to-all. Fine-tuning with vehicle losses alone changes the designated vehicle scores by at most 0.001. Cross-domain integration drives the vehicle gains, while joint supervision maintains all three tasks without a detectable vehicle-score penalty. We therefore favor directed connections for deployment, as they offer competitive vehicle forecasting performance with fewer connection parameters than all-to-all coupling.

Sections 4.2–4.3 now attribute AIDE gains to anchored kinematics and pretraining, vehicle-onset gains to cross-domain connections, and TTC-drop gains to structured road margins. Figure 4 and Appendix I separately evaluate vehicle information through frozen-feature readouts.

2. Fair comparison with openpilot.

Table 19 evaluates the surrogate’s attention-state predictions, rather than exact agreement with logged alerts. Our on-road study directly compares against openpilot’s actual driver-monitoring system, independently of this surrogate.

For a fair on-road comparison, we selected both systems’ operating thresholds on offline test recordings under the same nominal warning-budget target, then fixed them before the participant drives. For example, with a target of 20 warnings per hour, we adjust each system’s threshold on the same offline recordings to approach that rate, then keep both thresholds fixed during the participant drives.

Both systems therefore followed the same calibration rule. Under this protocol, TriDrive received higher paired ratings for appropriateness (+1.79) and timeliness (+2.67) (Section 4.4, Figure 5).

3. Study recordings and matched alert settings.

The 5.9 versus 2.5 hours describe the complete recording archive, including sessions before the questionnaire study and instrumented drives. They are not the paired driving durations used in the questionnaire comparison. Each participant drove the same route in their own vehicle under both conditions, with condition order alternated. The device logs also provide session-level driving durations, counting windows, and counts of presented level-2 prompts.

Both conditions used identical comma four devices and the same openpilot build. TriDrive reused openpilot’s own level-2 attention alert, so the text, display size, and alert sound were matched; level-3 escalation remained unchanged.

All the data, log and how we implemented TriDrive on comma is released on the Hugging Face repo: https://huggingface.co/AsianPlayer/TriDrive. You can also check in the supplimentary material. We will also make the full-modal data from the real-world vehicle testing (including driving video, driver video, audio, and CAN logs) publicly available after publication.

Thank you.

Dear Reviewer bdt1,

Thank you again for your constructive review. Our response is posted above, together with a revised PDF.

In brief, we added parameter-matched fusion and training-objective controls to clarify the source of the forecasting gains (Table 10). We also clarified the offline surrogate’s prediction targets (Table 19) and distinguished this comparison from the actual openpilot system used in the on-road study. Finally, we explained how the aggregate recording durations relate to the participant study, documented the matched alert presentation and threshold-calibration procedure, and provided links to the deployment materials and session logs.

Could you let us know whether these revisions address your concerns, or which points remain open? We will respond to any further questions before the discussion closes on November 18 (AoE). If the concerns are resolved, we would be grateful if you could reflect this in your assessment.

Best regards,

The authors

Reviewer_FViKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TriDrive, a unified framework that jointly forecasts driver kinematics, vehicle dynamics, and road demands through an automation-conditioned transition model. It argues that driver-state recognition alone cannot distinguish situations in which the same driver behavior may or may not warrant a warning, motivating joint understanding and prediction of driver, vehicle, and road states. TriDrive uses three separately pretrained temporal predictors, with directed residual connections through which driver and road context refine vehicle forecasts. To support on-vehicle deployment, the authors distill the road encoder and road-risk model into a compact student. The paper evaluates in-cabin kinematic forecasting on AIDE, cross-domain dynamics anticipation on 197.2 hours of BATON data, and a real-time contextual warning probe. Ablation studies examine directed residual coupling, modality-specific representations, and driver forecast features, showing improvements on selected forecasting targets. The authors also evaluate the warning probe using human-labeled warnings, report on-vehicle timing and input-validity results, and conduct a paired on-road study with 14 participants. The results demonstrate gains on selected forecasting targets and the feasibility of on-vehicle execution. However, the downstream benefits of joint forecasting for warning decisions have yet to be established, as the deployed warning probe uses current-state features and forecast features do not consistently improve offline warning AUROC.

Critical strengths

The paper extends its evaluation beyond offline forecasting benchmarks to on-vehicle deployment. The authors distill the road encoder and risk model into a compact student model, report on-vehicle runtime, end-to-end latency, and input-validity metrics, and conduct a paired on-road study with 14 participants comparing TriDrive with openpilot's driver-monitoring system. The study reports higher perceived warning appropriateness and timeliness, as well as increased annoyance. The authors also acknowledge that forecast features do not consistently improve offline warning AUROC and are therefore excluded from the deployed warning policy. These results document the feasibility of on-vehicle execution and provide an empirical assessment of user-perceived warning quality, while the downstream benefits of joint forecasting for warning decisions remain unverified.

Critical weaknesses
  1. The downstream value of joint forecasting for warning decisions remains unvalidated. The joint model shows gains on several forecasting targets, but these gains are not shown to improve the warning probe. The deployed system uses only current-state features, so the on-road user study does not establish whether incorporating forecasts into warning decisions would provide additional benefits.

  2. The methodological novelty and the necessity of the proposed coupling design are insufficiently established. TriDrive combines modality-specific representations and three separately pretrained temporal predictors, which are connected through directed residual connections for joint forecasting. For on-vehicle deployment, the authors further distill the road encoder and risk model into a compact student network. While the cross-domain forecasting formulation is meaningful, the evidence supporting the specific architectural contribution is less conclusive. The ablations show improvements on selected targets, but the all-to-all variant achieves comparable or better results on some tasks, despite using more connection parameters. Although directional coupling offers a more parameter-efficient design, the experiments do not establish a consistent performance advantage over alternative coupling structures. Consequently, the specific benefits of the proposed coupling design remain insufficiently characterized.

Questions for authors
  1. Clarification on the downstream value of forecasting. Adding forecast features does not consistently improve offline warning AUROC, and the restricted oracle analysis using ground-truth future features does not establish a clear gain. Could the authors clarify why forecast features provide limited additional benefit for warning decisions? This is important for assessing the practical value of the proposed forecasting mechanism.

  2. Clarification on the necessity of directed coupling. Alternative coupling structures achieve comparable or better results on some forecasting tasks. Given the reported three-seed comparisons and evaluation on additional drivers, could the authors clarify which improvements from directed residual coupling are consistent across seeds, held-out drivers, and key forecasting targets, and how these improvements are supported by the reported uncertainty estimates? This would help establish the specific contribution of the proposed coupling design relative to alternative structures.

  3. Clarification on real-world input reliability. Appendix G reports that full road-token-bank coverage reaches only 50.6% in the optimized third drive, compared with 84.3% in offline replay. Although the joint forecasting model continues to run on-vehicle, its outputs are logged rather than used by the deployed warning policy. Could the authors clarify how the joint model handles incomplete road-token banks and whether forecast availability or quality is affected under these conditions? This would help establish the reliability of the joint forecasting pipeline under actual driving conditions, independently of its downstream use in the warning policy.

Reviewer_xn8nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TriDrive, a unified framework for jointly forecasting driver behavior, vehicle dynamics, and road conditions to support context-aware driver monitoring. It predicts future binary state transitions, continuous vehicle states, and road representations. The authors evaluate the approach on the AIDE benchmark and naturalistic driving data, demonstrating improvements in selected forecasting tasks. They also deploy a separate lightweight warning classifier using current multimodal features and conduct a 14-participant real-world user study, reporting improved warning appropriateness and timeliness.

Critical strengths
  1. Incorporating driver, vehicle, and road context is well motivated for improving driver monitoring.
  2. The paper includes multiple datasets, ablation studies, real-world deployment, and a 14-participant user study. Experiments are comprehensive.
  3. The anchored kinematic representation design for driver motion forecasting is effective, and achieves competitive results on AIDE together with large-scale pose pretraining.
Critical weaknesses
  1. The necessity and fundamental benefits of jointly forecasting driver, vehicle, and road states are insufficiently established. The central motivation of the paper is that driver behavior should be understood in conjunction with vehicle dynamics and road conditions. While this motivation is reasonable, it primarily justifies multimodal context integration, rather than necessarily requiring the joint prediction of all three domains.
  2. The novelty is limited and justification for several architectural choices is insufficient. The method appears to be an aggregation of task-specific engineering choices rather than a framework derived from a clearly elaborated and experimentally validated modeling principle. It looks more engineering than academic.
  3. The presentation of the architecture is insufficiently clear, and several implementation details are not well connected to Figure 2. Section 3.2 states that the driver branch consumes five historical CAN signals as part of its 18-dimensional input. However, Figure 2 does not clearly illustrate this vehicle-to-driver input dependency.
  4. The proposed joint forecasting framework is not directly responsible for the reported improvements in downstream driver monitoring.
Questions for authors
  1. What specific limitations require joint forecasting rather than simpler multimodal fusion or independent prediction?
  2. How does the proposed joint forecasting framework contribute to the final driver monitoring performance?
1 comment

Dear Reviewer xn8n,

Thank you again for your review. Our response is posted above, together with a revised PDF.

In brief, we have added evidence showing how vehicle information improves predictive representations (Figure 4), matched fusion and training-objective controls to identify the source of the forecasting gains (Table 10), and a redrawn architecture making the CAN-to-driver input explicit (Figure 3). We also clarified the relationship between forecasting and the deployed current-state warning probe, and demonstrated anticipation gains that reproduce on 60 additional drivers (Table 11).

Could you let us know whether these revisions address your concerns, or which points remain open? We will respond to any further questions before the discussion closes on November 18 (AoE). If the concerns are resolved, we would be grateful if you could reflect this in your assessment.

Best regards,

The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

New experiment: forecasting anticipates driver inattention, and joint modeling improves it

Thank you. We added a new experiment that addresses this directly (pre-specified, three seeds, driver-cluster intervals, replicated on 60 additional drivers).

Forecasting anticipates driver transitions that the current state alone cannot reliably reveal. On the four validated driver endpoints, the default joint model forecasts hands leaving the wheel, the hands' return, and the head turning away above a label-free current-state heuristic (Table A).

Vehicle context further improves this anticipation in manual driving (new). Adding vehicle signals to the driver forecaster raises head-turn PR-AUC by +0.069, reproduced on the additional drivers (+0.067), and detects more onsets at least 1 s early at a 10 % anchor-level false-alarm budget set on validation (Table B). Every row in both tables meets our criterion.

The warning decision still reads the current state, as stated in our previous reply; the two functions run together on the vehicle (2.35 Hz forecasting alongside the 5 Hz probe).

Table A. Driver transitions: default joint model vs. a label-free current-state heuristic (PR-AUC@3 s)

EventJoint modelCurrent-state heuristic
Hands leave the wheel (manual)0.4360.271
Hands return to the wheel (manual)0.8260.691
Head turns away (manual)0.4740.323
Head turns away (assistance engaged)0.2850.238

Table B. New: driver forecaster with vs. without vehicle signals (manual driving)

MeasureDriversWithWithoutΔ
Head-turn PR-AUCTest0.4440.375+0.069
Head-turn PR-AUC60 additional0.4450.377+0.067
Hand-leave PR-AUCTest0.4260.411+0.015
Hand-leave PR-AUC60 additional0.4720.446+0.026
Hand-leave onsets detected ≥ 1 s earlyTest0.3770.351+0.026
Hand-leave onsets detected ≥ 1 s early60 additional0.3880.371+0.017
Head-turn onsets detected ≥ 1 s early60 additional0.4400.395+0.045

Detection rows: 10 % anchor-level false-alarm budget set on validation.

AuthorsOpenReview ↗

About concern "The proposed joint forecasting framework is not directly responsible for the reported improvements in downstream driver monitoring." New experiment: forecasting anticipates driver inattention, and vehicle context improves it.

Thank you. We added a new experiment that addresses this directly (pre-specified, three seeds, driver-cluster intervals, replicated on 60 additional drivers).

Forecasting anticipates driver transitions that the current state alone cannot reliably reveal. On the four validated driver endpoints, the default joint model forecasts hands leaving the wheel, the hands' return, and the head turning away above a label-free current-state heuristic (Table A).

Vehicle context further improves this anticipation in manual driving (new). Adding vehicle signals to the driver forecaster raises head-turn PR-AUC by +0.069, reproduced on the additional drivers (+0.067), and detects more onsets at least 1 s early at a 10 % anchor-level false-alarm budget set on validation (Table B). Every row in both tables meets our criterion.

The warning decision still reads the current state, as stated in our previous reply; the two functions run together on the vehicle (2.35 Hz forecasting alongside the 5 Hz probe).

Table A. Driver transitions: default joint model vs. a label-free current-state heuristic (PR-AUC@3 s)

EventJoint modelCurrent-state heuristic
Hands leave the wheel (manual)0.4360.271
Hands return to the wheel (manual)0.8260.691
Head turns away (manual)0.4740.323
Head turns away (assistance engaged)0.2850.238

Table B. New: driver forecaster with vs. without vehicle signals (manual driving)

MeasureDriversWithWithoutΔ
Head-turn PR-AUCTest0.4440.375+0.069
Head-turn PR-AUC60 additional0.4450.377+0.067
Hand-leave PR-AUCTest0.4260.411+0.015
Hand-leave PR-AUC60 additional0.4720.446+0.026
Hand-leave onsets detected ≥ 1 s earlyTest0.3770.351+0.026
Hand-leave onsets detected ≥ 1 s early60 additional0.3880.371+0.017
Head-turn onsets detected ≥ 1 s early60 additional0.4400.395+0.045

Detection rows: 10 % anchor-level false-alarm budget set on validation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.