acceptodds
Under review as a conference paper at ICLR 2027

DriveSkel-LM: LLM-Based Driving Route Generation via Road-Name-Aware Skeletons

Abstract

Recent work in transit planning shows that language models can generate structured routes from discrete network tokens. Extending this to driving is challenging because urban road networks are denser and more highly branched, and driving routes can contain long intersection sequences: decoding every intersection lengthens autoregressive generation, so an early error can cause missed road-change decisions, endpoint attachment failures, or disconnected routes. We introduce DriveSkel-LM, a road-name-aware framework for reliable driving route planning. Given a request, DriveSkel-LM predicts a compact skeleton of the origin, destination, and globally unique intersections at road-name changes, with aligned road names, preserving route-level decisions while omitting unchanged-road intersections. We adapt Qwen3-4B through intersection tokens and continued pre-training on anonymized navigation routes, road connectivity, and POI descriptions. Uniquely specified segments are reconstructed deterministically without graph search, while is used only for locally recoverable gaps. On temporally separated Beijing data, we evaluate DriveSkel-LM on optimal routing, road-name-constrained routing, and en-route search; on optimal routing it produces more reliable routes than full intersection-level generation, demonstrating the potential of LLM-based driving route planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, traffic reasoning, large language models, graph neural networks, scene understanding, embodied AI, decision explanation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AdU5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DriveSkel-LM, a road-name-aware LLM for reliable driving route planning. It predicts a compact skeleton with origin, destination and road-name-change intersections paired with road names, removing redundant intermediate nodes. Built on Qwen3-4B with custom intersection tokens and domain continued pre-training, the framework deterministically restores unique segments and uses A* only for local recoverable gaps. Evaluated on temporally split Beijing data over three routing tasks, it yields more reliable routes than full intersection decoding and validates LLM-based route planning potential.

Critical strengths
  1. The paper presents a well-motivated skeletonization design. By predicting only intersections where road names change instead of all GPS or intersection points, the method shortens autoregressive sequences and mitigates error accumulation, which is the key failure mode of prior TransitLM-style full-sequence route generation on dense driving road networks.
  2. The paper designs three practical routing tasks and adopts temporally split training and test data. Ablation experiments verify the importance of the skeleton representation for continued pre-training, dataset scale, and the trade-off between single-task and multi-task SFT. It also includes comparisons against general LLMs to demonstrate the necessity of domain continued pre-training and custom intersection tokens.
Critical weaknesses
  1. The paper only conducts evaluations on the Beijing dataset and does not validate whether the skeleton-based route generation paradigm can generalize to other cities with different road topology, naming conventions and intersection density. Could the authors provide experiments on multiple cities to verify the cross-city transferability of the proposed skeleton-based paradigm?
  2. The comparison against general LLMs is not a fair benchmark, as the metric definitions and output formats differ substantially. The paper should add more classical route planning baselines (e.g., A*, hierarchical routing) for comprehensive comparison on routing quality. Will the authors supplement comparisons with classic routing algorithms to enable fair and comprehensive evaluation of routing quality?
  3. The paper adopts a temporally split train-test setting which cannot guarantee generalization to unseen routes. Without deduplication, identical routes may appear in both training and test partitions, leaving ambiguity about whether the strong performance arises from memorizing repeated navigation routes instead of learning general routing decision rules. Can the authors clarify whether the promising performance mainly comes from memorization of repeated routes or learning general routing decision rules after deduplicating identical routes across train and test partitions?
  4. The paper motivates the road-name-aware skeleton partly by reducing autoregressive decoding length and restricting graph search to locally recoverable gaps. However, it does not report the average generated sequence length, end-to-end inference latency.
Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Four-City Operation, Classical Baselines, and Length Reported Apart from Latency

Thank you for recognizing the skeleton design and task setup. We address the four concerns with existing supplemental evidence, keeping its extended training and recovery setting distinct from the submitted Beijing experiment.

W1: cross-city evaluation. We report four trained cities under the same supplemental evaluator, including every request in each city.

Citybest_routeroad_namePOI
Beijing269/277 (97.1%)217/318278/305
Shanghai305/318 (95.9%)198/287234/254
Chengdu227/238 (95.4%)148/239175/216
Shenzhen166/167 (99.4%)116/156212/225

The bold column is the best-route composite and the other two are the task-specific composites. These results support operation across the four networks without changing the evaluation pipeline between cities.

The scope is trained-city operation: the recovery permits repairs of F intervals beyond the paper's fallback, accepts unverified shared-name chords and can stop early, so the composite is not a certified complete-route success rate. Original Beijing vocabulary/data figures are preserved, and the deployed four-city token inventory is not inferred from them. Single-/multi-task differences also include optimization budgets and are not attributed solely to task mixing. All four cities come from one static snapshot; these boundaries separate multi-city evaluation in trained cities from zero-shot city transfer or temporal graph updates.

W2: classical routing and fair comparison. We add Dijkstra on best_route, with the link-end intersection IDs as origin and destination, so no parsing or endpoint-selection error enters; it minimizes haversine-weighted distance on the same directed graph.

best_route, 980 common recordsDriveSkel-LM + recoveryDijkstra
Agreement with the reconstructed reference51.94%20.86%
Summed length / reported label mileage0.9650.849
Mileage relative to Dijkstra1.1371.000

Dijkstra finds paths on 999/999 evaluated requests, with one missing evaluation among 1,000.

On the 980 common records both sides are scored against the same reconstructed reference, and Dijkstra is the shortest path on that same directed graph. DriveSkel-LM + recovery agrees with the reference on 51.94% of requests against 20.86%, and its summed length is 0.965 of the reported label mileage against 0.849, so it is the closer of the two on both measures.

W3: repeated routes. The audit quantifies overlap and re-scores without it.

SFT audit (task order best_route/road_name/POI)best_routeroad_namePOI
Skeleton matches per 1,0001111
Road-description matches3115
Requests retained after the union951948947
Composite after / before filtering96.6 / 96.767.0 / 67.989.9 / 89.9
CPT ODs covered999998996
CPT ordered-skeleton matches in content_2449197

The evaluation requests come from the held-out _sample_10000 split. Matching them against _train gives 0/1/0 OD matches and 1/11/1 skeleton matches per 1,000, and removing every matched request moves the composite from 96.7/67.9/89.9 to 96.6/67.0/89.9, so the SFT tables carry no leakage into the reported results.

The same ODs also appear in the CPT knowledge text (999/998/996 covered), with 44/91/97 of the 1,000 routes present as ordered skeletons; removing those leaves 96.9/68.0/89.6 on 956/909/903 requests, so this channel does not carry the results either. The two audits are separate; alternatives in content_1 plus record linkage limit coverage, so the paper's temporal-split qualification stands.

W4: output length and recovery cost. The emitted-length audit reports the observed output behaviour.

MeasureSkeleton (best_route / road_name / POI)Full-intersection
Generated IDs, mean13.29 / 13.41 / 12.131,805.73
Generated IDs, median13 / 13 / 122,046
Generated IDs, P9522 / 22 / 192,046
Reference IDs, median≈13 (ratio 1.01–1.04)36
Unclosed outputs1 / 0 / 0 per 1,00091.7%
Recovery calls, mean2.10 / 2.05 / 1.71 ms—

These quantify the output behaviour of the recorded decoders; different batches and checkpoints do not isolate the cause of the gap.

The recovery-timing measurement gives the saved recovery-call times above, which exclude generation, graph loading, classification and service overhead. The numbers above are in the units that were measured: intersection IDs for length, single recovery calls for time. Character-based token estimates are not used as measured compression factors, and neither ID counts nor recovery-only timings are converted into end-to-end speedups; end-to-end service latency is outside what these logs record.

Thank you for rebuttal. However, authors have not fully addressed my concerns. For W1, could the authors provide experiments on cross-city transferability to unseen cities, i.e., training on city A and testing on city B, or provide an analysis that explicitly clarifies the generalization limitations of DriveSkel-LM? For W4, in driving route planning, there are also high requirements for real-time performance. Since this paper uses an LLM, have the authors considered the issue of inference efficiency? The authors should report the inference overhead of DriveSkel-LM and other baselines and provide a comparative analysis. If the computational overhead is too high, how do the authors address this issue? After all, i think efficiency appears to be a fundamental requirement in route planning with high real-time demands.

AuthorsOpenReview ↗

Trained-City Scope and Inference Latency against Dijkstra, Single-Pass, and Agent Baselines

Thank you for the follow-up. We clarify the limits of our cross-city evaluation and report inference latency against baselines performing the same tasks.

W1. Cross-city applicability and generalization limits

DriveSkel-LM represents intersections with dedicated tokens. Training only on city A does not ground city B’s intersection tokens in its roads, connectivity, and POIs. Adding new tokens without training does not provide the knowledge needed for route planning. Access to city B’s graph during recovery does not resolve this limitation, since recovery operates on decisions already generated by the model.

Our multi-city experiments train on multiple cities and evaluate each city separately. They test whether the same representation and training procedure work across different networks, rather than whether zero-shot routing works in a city absent from training. We will state this distinction explicitly. Deployment in a new city currently requires vocabulary extension and local training data; reducing this adaptation requirement remains future work.

W4. Inference efficiency

We now report end-to-end latency. All LLM-based methods use the same GPU hardware and inference configuration.

For unconstrained routing, we compare with Dijkstra and full-intersection generation.

TaskMethodLatency P50 (ms)Latency P95 (ms)
UnconstrainedDijkstra, routing only33173
UnconstrainedFull-intersection LLM≈57,000≈61,000
UnconstrainedDriveSkel-LM2,8373,794

Full-intersection timings are computed on outputs truncated at the generation limit. DriveSkel-LM is substantially slower than Dijkstra, and we do not claim an efficiency advantage over shortest-path search.

Road and POI requests additionally require language interpretation and entity retrieval. We therefore evaluate complete parsing, retrieval, and constrained-routing pipelines. Timings include parsing or generation, retrieval where applicable, and routing or recovery.

We report success-weighted IoU (SW-IoU): each request contributes its final-route IoU if it passes every complete-route check, and zero otherwise, averaged over all requests.

TaskMethodSW-IoU (%)P50 latency summary (ms)P95 latency summary (ms)
RoadSingle-pass parsing + retrieval + routing23.16281,256
RoadIterative parsing + retrieval + routing (agent-based)65.83,9435,418
RoadDriveSkel-LM48.52,8993,850
POISingle-pass parsing + retrieval + routing15.26011,144
POIIterative parsing + retrieval + routing (agent-based)43.56,4688,495
POIDriveSkel-LM55.22,8003,516

DriveSkel-LM offers a favorable quality–latency trade-off. On POI requests, it achieves higher SW-IoU than the multi-turn agent (55.2% vs. 43.5%) with substantially lower reported latency. On Road requests, it trades lower SW-IoU for faster responses. Compared with the single-pass pipeline, it achieves higher SW-IoU on both tasks at greater latency. Its single-generation workflow also avoids repeated retrieval and routing calls, simplifying inference orchestration.

For conversational road and POI recommendation, DriveSkel-LM returns results in 2.8-2.9 seconds at P50 and 3.5-3.9 seconds at P95. These results support direct route generation as a practical alternative worth developing. We will revise the paper to clarify its generalization limits and report this balance between route performance and latency, with further improvements focused on complete-route success.

Reviewer_YdgeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DriveSkel-LM generates a compact route skeleton containing endpoints and intersections where the road name changes. It adds intersection tokens to Qwen3-4B and trains on navigation records, connectivity descriptions, and POI information. A static road graph expands the predicted skeleton, with constrained A* search used for recoverable gaps. The experiments cover unconstrained routing, road-name constraints, and en-route POI search on temporally separated Beijing data. Skeleton generation improves endpoint attachment substantially over full-intersection generation, but complete-route reliability remains difficult to assess from the reported metrics.

Critical strengths

-The representation is sensible. Road names alone are ambiguous, while predicting every intersection creates unnecessarily long sequences. Using road-change intersections provides a reasonable middle ground.

-The recovery procedure deliberately refuses to repair arbitrary gaps. This makes failures visible instead of allowing unrestricted graph search to compensate for any generated skeleton.

Critical weaknesses

-My main concern is that the evaluation does not clearly answer how often the system returns a usable route. Route Connectivity is measured per skeleton interval, and Both-End Attachment allows a 500 m discrepancy without checking legal access. Endpoint intersection IDs are also provided in the prompt. The road-constrained task reports 65.1% joint compliance, but only among road-name-consistent outputs. A common, unconditional route-level success metric is needed, especially since the length and overlap metrics exclude failed recoveries.

-The relevant routing baselines are missing. The general-purpose LLMs use different outputs and metrics, so their results do not show that DriveSkel-LM is the better route planner. Since the method already uses a road graph, a language parser followed by conventional constrained routing is an important comparison.

-The full-intersection baseline's 7.6% endpoint attachment is unusually poor and needs investigation. The paper does not separate errors caused by long-sequence prediction from truncation, repeated-node loops, or stopping behavior. Generation budgets and output-length statistics would help make this comparison interpretable.

-It is unclear how much information the skeleton loses before any learning error occurs. Same-name branches can make recovery ambiguous, yet no oracle-skeleton result is reported. The uniqueness check also needs a more concrete description. In addition, Equation 5 defines P using the two-name condition, whereas the following text calls that condition only necessary.

-The experimental setup is difficult to reproduce or assess for memorization. It uses one static graph, does not remove repeated routes across daily partitions, and omits basic optimization and decoding settings. The inability to release the data makes those missing details more consequential.

Questions for authors

-What fraction of requests in each task yields a connected route with valid endpoint access and all requested constraints satisfied, counting every test request?

-How does DriveSkel-LM compare with a language-to-constraints parser followed by graph routing?

-What are the decoding budgets, truncation rates, loop rates, and sequence lengths for the full-intersection and skeleton models?

-How well does recovery work with ground-truth skeletons, and how do the learned results change after repeated routes are removed across splits?

5 comments
AuthorsOpenReview ↗

An Unconditional Route-Level Metric, a Decode Audit, and a Bounded Uniqueness Check

Thank you. Task order is best_route/road_name/POI; all results use an extended setting, not the submitted Beijing experiment.

W1/Q1: request-level success and endpoints. The saved evaluator gives 96.7/67.9/89.9% over all 1,000 requests per task, failures counted as zero. It operates at the text and skeleton level rather than on the complete recovered route: it permits repairs, access reads skeleton endpoints, Road and POI compare text or name/category rather than passage, and parse alignment is only partly checked. These are diagnostic counts; the paper's usability claim rests on Table 2.

Indicatorbest_routeroad_namePOI
Joint success, failures = 096.7%67.9%89.9%
Directed-edge checks passed972979985
Endpoints within 500 m986992986
Supplied endpoint IDs copied970972959

The exact-copy row separates endpoint preservation from route generation.

W2/Q2: conventional routing. We compare best_route. Dijkstra receives the link-end intersection IDs as origin and destination, so no parsing or endpoint-selection error enters, and it minimizes haversine-weighted distance on the same directed graph.

best_route, 980 common recordsDriveSkel-LM + recoveryDijkstra
Agreement, reconstructed reference51.94%20.86%
Mileage relative to Dijkstra1.1371.000

Dijkstra returns paths on 999/999 evaluated requests. Both sides use the same reconstructed reference and the same directed graph. DriveSkel-LM + recovery matches the annotated route on 51.94% against 20.86%, at 1.137 of Dijkstra's mileage, following the labelled route rather than the shortest path.

W3/Q3: full-intersection diagnosis. The 1,000-response audit records a nominal 4,096-token budget.

MeasureSkeleton (best_route / road_name / POI)Full-intersection
Generated IDs, mean13.29 / 13.41 / 12.131,805.73
Generated IDs, median13 / 13 / 122,046
Reference IDs, median≈13 (ratio 1.01–1.04)36
Unclosed outputs1 / 0 / 0 per 1,00091.7%
Never emits destination—90.3%

Median unique/total IDs is 0.655 and the revisit gap 9 steps (P95 35); the 60.5% cap-hit figure is inferred from ID count, not finish reasons. These show repetition, destination omission and length, not the cause. Skeleton inference budget and loop rate are outside this audit, and a training cutoff is not a decoding budget; the batches differ, so replication is not exact.

W4/Q4: oracle recovery and Eq. (5). Under the same visualization recovery:

MeasureLabel skeletonPredicted skeleton
No recorded recovery failure1,000 / 1,000 / 1,000980 / 991 / 990
Non-adjacent pairs with multiple relaxed paths29.2 / 28.5 / 30.5%not applicable

This is a recovery diagnostic with permissive repairs and early stopping, so it reports recovery rate and constraint agreement. With no original dense reference, the measured quantity is recoverability; exact reconstruction would need that reference. The relaxed enumeration drops the required road name, skips direct neighbours and is bounded, so its search space is smaller than named-triplet uniqueness.

For unresolved intervals, kappa_a<=2 is necessary and sufficient for a path containing a with at most two road names, not uniqueness or label reproduction; we correct the sentence after Eq. (5). The saved code path applies that adjacency bound alone, without U's uniqueness test or the hard named-road constraint, and the configuration called paper, with its skel_fly filter, differs from the paper's setting.

W5/Q4: memorization and reproducibility. The overlap audit:

Overlap auditbest_route / road_name / POI
Requests retained after all-key matching951 / 948 / 947
Composite after / before filtering96.6 / 67.0 / 89.9% vs 96.7 / 67.9 / 89.9%
CPT ODs / ordered-skeleton matches999 / 998 / 996 ; 44 / 91 / 97

The evaluation requests come from the held-out _sample_10000 split. Removal leaves the CPT-unseen subset at 96.9/68.0/89.6 on 956/909/903 requests, and the SFT-filtered values stay within 1.0 point of the full set, so neither table carries leakage into these results. content_1 alternatives limit claims about unseen routes, though the audited rows are the rows this checkpoint trained on.

Training configuration:

SettingValue
CPT / SFT examples per task3,191,100 / 10,000
CPT init step / SFT steps (multi / single) / batch5,000 / 1,000 / 400 / 256
Precision / sharding / cutoff / LR / warm-up / decodingbf16 / ZeRO-3 / 2,048 / 2e-5 cosine / 15 / greedy

Runs are not step-matched; table sizes give row counts, not unique samples, seeds or vocabulary coverage. The paper's data restrictions remain.

Thank you for the additional results! The decoding and overlap analyses are helpful! However, it is still a bit unclear to me how often the system returns a complete, valid route satisfying all the constraints. The joint-success metric checks text and skeletons, and the new oracle results use a more permissive recovery procedure than the paper describes.

Could you report complete-route success over all requests for both predicted and ground-truth skeletons using the paper’s specified recovery procedure? The comparison with a language parser followed by constrained graph routing is also still missing, as the Dijkstra comparison only covers the unconstrained task. These remain my main concerns.

AuthorsOpenReview ↗

Complete-Route Success under the Paper's Recovery Rule, Constrained-Routing Baselines, and Per-Request Call Counts

Thank you for the follow-up. We have added complete-route evaluation under the paper’s recovery procedure and a comparison with language parsing, retrieval, and constrained routing.

W1. Complete-route success

We evaluate ground-truth and predicted skeletons on all 1,000 requests per task using Section 3.4’s recovery procedure. The evaluator now includes the uniqueness check and the hard named-road search constraint missing from our first response, and rejects intervals outside the permitted recovery scope. Success requires a complete directed path, valid endpoint access, and satisfaction of the request on that path. Any failed check counts as a failure; no requests are excluded.

Skeleton sourceUnconstrained success (%)Road success (%)POI success (%)
Ground-truth97.297.096.6
DriveSkel-LM78.461.275.7

Ground-truth success is below 100% mainly because inconsistent road naming and omitted intermediate road changes cause some intervals to exceed the two-road-name recovery budget. Additional failures arise from endpoint checks and road-constraint satisfaction.

Our first response reported higher composite scores using broader recovery and less stringent checks. To separate these effects, we also evaluate broader recovery with the same complete-path checks. Allowing search beyond the two-road-name limit raises unconstrained success from 78.4% to 95.5% on the same checkpoint and requests. These additional routes pass the same validity checks, but require recovery beyond the procedure specified in the paper.

W2. Language parsing, retrieval, and constrained routing

The parser extracts the road name and the take/avoid intent, or the POI name/category. Retrieval finds the corresponding road segments or POI access locations, and constrained routing computes a route satisfying those requirements.

For required roads, the router minimizes distance over paths containing a matched directed road segment; for avoided roads, it removes the matched edges before routing. For POI requests, it routes through a retrieved POI’s graph access location.

The single-pass variant runs this process once. The iterative variant (agent-based) uses previous tool results to revise its interpretation and retrieved candidates, stopping when the LLM judges the request satisfied. Final success is assessed by the evaluator, not by the LLM.

All methods use the same 1,000 requests per task and the same evaluation protocol. IoU follows Section 4.2 and is averaged over successfully recovered routes.

MethodRoad success (%)Road IoU (%)Road LLM callsRoad search & routing callsPOI success (%)POI IoU (%)POI LLM callsPOI search & routing calls
Single-pass parsing + retrieval + routing89.525.81186.417.611
Iterative parsing + retrieval + routing (agent-based)96.568.15.335.1698.844.15.867.51
DriveSkel-LM61.256.71075.759.210

All call counts are per-request averages. Search & routing calls count external tool invocations; DriveSkel-LM’s local graph recovery is not included.

The routing baselines explicitly enforce connectivity and the resolved constraints during graph search, which helps explain their higher success rates. DriveSkel-LM instead predicts route decisions directly and uses bounded local recovery, so generation errors outside that recovery scope lead to failures. Our current results show a clear disadvantage in complete-route success.

IoU differs by task: the iterative pipeline is higher on Road (68.1% vs. 56.7%), while DriveSkel-LM is higher on POI (59.2% vs. 44.1%). These IoU results concern successfully recovered routes, whereas success rates include all requests.

We will revise the paper to clarify that our goal is to explore direct route generation with bounded local recovery, rather than claim superiority over language parsing followed by constrained routing (including agent-based variants). These baselines clarify where our approach works well and where it falls short, with improving complete-route success as a priority for future work.

Reviewer_XCL8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts token-based route generation to driving. Instead of emitting every intersection, DriveSkel-LM emits a skeleton origin, destination and the intersections at which the road name changes, each paired with the road name traversed. Qwen3-4B is extended with 49,082 intersection tokens, continued-pretrained on ~3.2M Beijing records, and fine-tuned per task. Relative to full-intersection decoding, the skeleton raises both-end attachment from 7.6% to 95.2% and cuts the route-length ratio from 2.93 to 1.21. Ablations cover CPT representation, CPT volume and single- vs multi-task SFT; six general-purpose LLMs are evaluated on a relaxed road-name-only task.

Critical strengths
  • The skeleton representation is a sensible middle ground between road-name-only and per-segment vocabularies, and the U/P/F recovery partition (Eq. 4–6) is clearly formalised, including an honest statement of when local recovery is not guaranteed.

  • Large real navigation corpus with temporal train/test separation.

Critical weaknesses
  • No comparison with conventional routing. Recovery already reads the static road graph at inference, so a plain graph router such as Dijkstra or A* over the same graph (or a hierarchical one such as contraction hierarchies) is the essential baseline.

  • Generalisation issue: There is only one city, one graph snapshot, and no train/test de-duplication. The label is “the route the user actually executed”, which is one of many acceptable routes, so IoU/SR against a single reference is an ambiguous target.

Questions for authors
  • How much of the test set consists of origin–destination pairs or complete skeletons already seen in CPT/SFT data, and how do the metrics change on the subset of entirely unseen routes? Since the vocabulary is 49k Beijing-specific intersection tokens, could you also comment on how the approach would transfer to a new city or to a road-graph update ?
3 comments
AuthorsOpenReview ↗

Graph-Routing Baselines, Training-Overlap Audits, and Four-City Operation

Thank you for recognizing the skeleton representation and for the suggestions on baselines and generalization. We address them with the existing supplemental evaluations, distinguishing their extended setting from the submitted Beijing experiment.

Conventional routing. We compare Dijkstra on best_route. Both methods use the same directed graph, and Dijkstra minimizes the sum of haversine edge distances.

MeasureDriveSkel-LM + recoveryDijkstraDenominator
Reference agreement (RA-IoU)51.94%20.86%980 with both scores
Same, missing scores and failed recoveries as zero50.90%20.89%all 1,000
Summed filled length / summed reported label length0.9650.849980
Mileage relative to Dijkstra1.1371.000980
Requests returning a path1,000999 (one has no saved evaluation)all 1,000

This comparison separates agreement with the annotated choice from distance optimality. The 980 common requests are scored against the same reconstructed reference, and Dijkstra is the shortest path on the same directed graph from the endpoint IDs the prompt supplies, so it receives no information the model did not have. DriveSkel-LM + recovery agrees with that reference on 51.94% against 20.86%, and its summed length is 0.965 of the reported label mileage against 0.849, so it is the closer of the two on both measures. A* and hierarchical routing target the same graph-cost optimum, although tied routes can differ, so the comparison is on route choice.

Training overlap and filtered performance. The audit reports overlap and the performance that remains after the detected matches are removed.

SFT audit (task order best_route/road_name/POI)best_routeroad_namePOI
Skeleton-ID matches per 1,0001111
Road-description matches3115
OD matches0110
Requests retained after the union951948947
Composite after / before filtering96.6 / 96.767.0 / 67.989.9 / 89.9
CPT ODs covered999998996
CPT ordered-skeleton matches in content_2449197

The evaluation requests come from the held-out _sample_10000 split. Every retained request stays in the denominator, and removing the matched requests moves the composite from 96.7/67.9/89.9 to 96.6/67.0/89.9; on the CPT side the same removal leaves 96.9/68.0/89.6 on 956/909/903 requests, so neither table carries leakage into the reported results.

The SFT and CPT audits are separate, and alternatives in content_1 plus record-linkage discrepancies limit coverage, so the filtered subsets are described by their matching rules rather than as entirely unseen routes. The 1/2/4 unmatched ODs are too few for an unseen-OD estimate. The multi-task checkpoint reads a differently named table (multi_task_ix_sft_skeleton_5w_0927_bssc_train) but the same two prompt and label columns: its 120,000 (prompt, label) pairs are exactly the 120,000 rows of the three audited tables, and the request ids agree in both directions, so the audit covers the data that model trained on.

Other cities and graph updates. The same supplemental pipeline is evaluated in four trained cities.

Citybest-route compositeRate
Beijing269/27797.1%
Shanghai305/31895.9%
Chengdu227/23895.4%
Shenzhen166/16799.4%

This supports operation across those networks under one implementation. The original 49,082-token vocabulary describes Beijing; new-city deployment requires appropriate intersection mappings and adaptation data. For graph updates, stable intersection IDs can be retained while new/removed IDs and changed connectivity are refreshed, followed by targeted training. The four-city evidence comes from one static snapshot and concerns trained-city operation rather than zero-shot transfer, temporal graph updates or measured update cost.

One reference is not the only correct route. IoU and skeleton recall are reference-agreement diagnostics, while RLR measures length relative to the reference and is not lower-is-better. The supplemental composite is an implementation-level diagnostic: it checks skeleton access, permits repairs of F intervals beyond the paper's fallback, accepts unverified shared-name chords and allows early stopping, so it does not certify complete-route usability. The paper's specified recovery and original figures remain separate from this extended evaluation, and alternative valid routes are not treated as errors.

Reviewer_XCL8OpenReview ↗

Reply to the rebuttal

Thanks for the detailed response and extra results. The new Dijkstra baseline, train-test overlap check, and four-city experiments address most of my main concerns. Please remember to include these new baselines and discussions in the revised paper. I have raised my score accordingly.

Thank you for the feedback. We are glad the new results address the main concerns, and we will include all corresponding baselines, audits, and discussions in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.