- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DriveSkel-LM, a road-name-aware LLM for reliable driving route planning. It predicts a compact skeleton with origin, destination and road-name-change intersections paired with road names, removing redundant intermediate nodes. Built on Qwen3-4B with custom intersection tokens and domain continued pre-training, the framework deterministically restores unique segments and uses A* only for local recoverable gaps. Evaluated on temporally split Beijing data over three routing tasks, it yields more reliable routes than full intersection decoding and validates LLM-based route planning potential.
Critical strengths
- The paper presents a well-motivated skeletonization design. By predicting only intersections where road names change instead of all GPS or intersection points, the method shortens autoregressive sequences and mitigates error accumulation, which is the key failure mode of prior TransitLM-style full-sequence route generation on dense driving road networks.
- The paper designs three practical routing tasks and adopts temporally split training and test data. Ablation experiments verify the importance of the skeleton representation for continued pre-training, dataset scale, and the trade-off between single-task and multi-task SFT. It also includes comparisons against general LLMs to demonstrate the necessity of domain continued pre-training and custom intersection tokens.
Critical weaknesses
- The paper only conducts evaluations on the Beijing dataset and does not validate whether the skeleton-based route generation paradigm can generalize to other cities with different road topology, naming conventions and intersection density. Could the authors provide experiments on multiple cities to verify the cross-city transferability of the proposed skeleton-based paradigm?
- The comparison against general LLMs is not a fair benchmark, as the metric definitions and output formats differ substantially. The paper should add more classical route planning baselines (e.g., A*, hierarchical routing) for comprehensive comparison on routing quality. Will the authors supplement comparisons with classic routing algorithms to enable fair and comprehensive evaluation of routing quality?
- The paper adopts a temporally split train-test setting which cannot guarantee generalization to unseen routes. Without deduplication, identical routes may appear in both training and test partitions, leaving ambiguity about whether the strong performance arises from memorizing repeated navigation routes instead of learning general routing decision rules. Can the authors clarify whether the promising performance mainly comes from memorization of repeated routes or learning general routing decision rules after deduplicating identical routes across train and test partitions?
- The paper motivates the road-name-aware skeleton partly by reducing autoregressive decoding length and restricting graph search to locally recoverable gaps. However, it does not report the average generated sequence length, end-to-end inference latency.
Questions for authors
See weakness
3 comments
Thank you for rebuttal. However, authors have not fully addressed my concerns. For W1, could the authors provide experiments on cross-city transferability to unseen cities, i.e., training on city A and testing on city B, or provide an analysis that explicitly clarifies the generalization limitations of DriveSkel-LM? For W4, in driving route planning, there are also high requirements for real-time performance. Since this paper uses an LLM, have the authors considered the issue of inference efficiency? The authors should report the inference overhead of DriveSkel-LM and other baselines and provide a comparative analysis. If the computational overhead is too high, how do the authors address this issue? After all, i think efficiency appears to be a fundamental requirement in route planning with high real-time demands.
Trained-City Scope and Inference Latency against Dijkstra, Single-Pass, and Agent Baselines
Thank you for the follow-up. We clarify the limits of our cross-city evaluation and report inference latency against baselines performing the same tasks.
W1. Cross-city applicability and generalization limits
DriveSkel-LM represents intersections with dedicated tokens. Training only on city A does not ground city B’s intersection tokens in its roads, connectivity, and POIs. Adding new tokens without training does not provide the knowledge needed for route planning. Access to city B’s graph during recovery does not resolve this limitation, since recovery operates on decisions already generated by the model.
Our multi-city experiments train on multiple cities and evaluate each city separately. They test whether the same representation and training procedure work across different networks, rather than whether zero-shot routing works in a city absent from training. We will state this distinction explicitly. Deployment in a new city currently requires vocabulary extension and local training data; reducing this adaptation requirement remains future work.
W4. Inference efficiency
We now report end-to-end latency. All LLM-based methods use the same GPU hardware and inference configuration.
For unconstrained routing, we compare with Dijkstra and full-intersection generation.
| Task | Method | Latency P50 (ms) | Latency P95 (ms) |
|---|---|---|---|
| Unconstrained | Dijkstra, routing only | 33 | 173 |
| Unconstrained | Full-intersection LLM | ≈57,000 | ≈61,000 |
| Unconstrained | DriveSkel-LM | 2,837 | 3,794 |
Full-intersection timings are computed on outputs truncated at the generation limit. DriveSkel-LM is substantially slower than Dijkstra, and we do not claim an efficiency advantage over shortest-path search.
Road and POI requests additionally require language interpretation and entity retrieval. We therefore evaluate complete parsing, retrieval, and constrained-routing pipelines. Timings include parsing or generation, retrieval where applicable, and routing or recovery.
We report success-weighted IoU (SW-IoU): each request contributes its final-route IoU if it passes every complete-route check, and zero otherwise, averaged over all requests.
| Task | Method | SW-IoU (%) | P50 latency summary (ms) | P95 latency summary (ms) |
|---|---|---|---|---|
| Road | Single-pass parsing + retrieval + routing | 23.1 | 628 | 1,256 |
| Road | Iterative parsing + retrieval + routing (agent-based) | 65.8 | 3,943 | 5,418 |
| Road | DriveSkel-LM | 48.5 | 2,899 | 3,850 |
| POI | Single-pass parsing + retrieval + routing | 15.2 | 601 | 1,144 |
| POI | Iterative parsing + retrieval + routing (agent-based) | 43.5 | 6,468 | 8,495 |
| POI | DriveSkel-LM | 55.2 | 2,800 | 3,516 |
DriveSkel-LM offers a favorable quality–latency trade-off. On POI requests, it achieves higher SW-IoU than the multi-turn agent (55.2% vs. 43.5%) with substantially lower reported latency. On Road requests, it trades lower SW-IoU for faster responses. Compared with the single-pass pipeline, it achieves higher SW-IoU on both tasks at greater latency. Its single-generation workflow also avoids repeated retrieval and routing calls, simplifying inference orchestration.
For conversational road and POI recommendation, DriveSkel-LM returns results in 2.8-2.9 seconds at P50 and 3.5-3.9 seconds at P95. These results support direct route generation as a practical alternative worth developing. We will revise the paper to clarify its generalization limits and report this balance between route performance and latency, with further improvements focused on complete-route success.
Four-City Operation, Classical Baselines, and Length Reported Apart from Latency
Thank you for recognizing the skeleton design and task setup. We address the four concerns with existing supplemental evidence, keeping its extended training and recovery setting distinct from the submitted Beijing experiment.
W1: cross-city evaluation. We report four trained cities under the same supplemental evaluator, including every request in each city.
The bold column is the best-route composite and the other two are the task-specific composites. These results support operation across the four networks without changing the evaluation pipeline between cities.
The scope is trained-city operation: the recovery permits repairs of F intervals beyond the paper's fallback, accepts unverified shared-name chords and can stop early, so the composite is not a certified complete-route success rate. Original Beijing vocabulary/data figures are preserved, and the deployed four-city token inventory is not inferred from them. Single-/multi-task differences also include optimization budgets and are not attributed solely to task mixing. All four cities come from one static snapshot; these boundaries separate multi-city evaluation in trained cities from zero-shot city transfer or temporal graph updates.
W2: classical routing and fair comparison. We add Dijkstra on best_route, with the link-end intersection IDs as origin and destination, so no parsing or endpoint-selection error enters; it minimizes haversine-weighted distance on the same directed graph.
Dijkstra finds paths on 999/999 evaluated requests, with one missing evaluation among 1,000.
On the 980 common records both sides are scored against the same reconstructed reference, and Dijkstra is the shortest path on that same directed graph. DriveSkel-LM + recovery agrees with the reference on 51.94% of requests against 20.86%, and its summed length is 0.965 of the reported label mileage against 0.849, so it is the closer of the two on both measures.
W3: repeated routes. The audit quantifies overlap and re-scores without it.
The evaluation requests come from the held-out
_sample_10000split. Matching them against_traingives 0/1/0 OD matches and 1/11/1 skeleton matches per 1,000, and removing every matched request moves the composite from 96.7/67.9/89.9 to 96.6/67.0/89.9, so the SFT tables carry no leakage into the reported results.The same ODs also appear in the CPT knowledge text (999/998/996 covered), with 44/91/97 of the 1,000 routes present as ordered skeletons; removing those leaves 96.9/68.0/89.6 on 956/909/903 requests, so this channel does not carry the results either. The two audits are separate; alternatives in content_1 plus record linkage limit coverage, so the paper's temporal-split qualification stands.
W4: output length and recovery cost. The emitted-length audit reports the observed output behaviour.
These quantify the output behaviour of the recorded decoders; different batches and checkpoints do not isolate the cause of the gap.
The recovery-timing measurement gives the saved recovery-call times above, which exclude generation, graph loading, classification and service overhead. The numbers above are in the units that were measured: intersection IDs for length, single recovery calls for time. Character-based token estimates are not used as measured compression factors, and neither ID counts nor recovery-only timings are converted into end-to-end speedups; end-to-end service latency is outside what these logs record.