acceptodds
Under review as a conference paper at ICLR 2027

ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligence

16 views

Abstract

Training a spatial VLM to answer which side an object lies on does not require it to preserve the metric displacement that determines that answer, while supervision for a distance estimate may leave categorical relations largely unconstrained. When post-training emphasizes one readout, improvements can therefore concentrate on that interface while transfer to others remains weak or even degrades; we call this observed specialization interface lock-in. ChronoReplay addresses it by replaying heterogeneous spatial supervision as chronologically ordered episodes, retaining source-native targets, and balancing supervision across families. Under the same observation access, a single ChronoReplay checkpoint improves OVO-S-Bench, OST-Bench, and UCS-Bench together, including a 13.41-point gain on OST-Bench, and achieves the highest three-benchmark mean among the supervision settings compared here. Same-data controls further separate the ingredients of this transfer: answer serialization redistributes gains across OST and UCS, family-balanced learning improves over ordinary joint SFT and natural turn weighting, and chronological replay exceeds a shifted-order control by about 2.41 OST points across two paired seeds. Adaptive frame selection then changes evidence access while keeping the trained model fixed, providing an additional inference-time gain at lower visual cost. **Streaming spatial performance depends on what distinctions supervision preserves, how heterogeneous histories are organized during learning, and what causal evidence inference makes available at test time.**

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Spatial Intelligence, Vision-Language Models, Spatial Reasoning, Temporal-Origin Alignment, Multimodal Post-Training, Cross-Interface Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6xeBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines single-turn and multi-turn spatial question-answering data, keeps every target in its source-native answer format, preserves each answer's preceding observations and interaction history, and normalizes and re-weights the training losses of the supervision families to improve performance across spatial benchmarks; a separate inference-time frame-selection policy is evaluated on OVO-S.

Critical strengths

Two critical strengths are listed below.

  1. The comparison in Tables 9–10 keeps the training data, histories, and budget fixed. Giving categorical, numerical, and trajectory supervision equal total loss weights improves on the natural-turn-weighting baseline, which already averages losses within each answer, across all three benchmarks and each of three paired seeds. This supports a concrete benefit from the chosen supervision weights.

  2. The paper first compares the base and trained models under the same observations, then changes frame selection while keeping the trained model fixed. The two OVO-S gains are 4.91 and 4.09 points, respectively. This helps distinguish improvements from training from those obtained by selecting more suitable observations.

Critical weaknesses

Three critical weaknesses are listed below.

  1. The gains after joint training do not by themselves establish the cross-interface transfer suggested by the title and keywords, although the paper itself refrains from attributing the joint gain to a single mechanism. The full model already receives categorical, numerical, and temporal supervision, including 2,545 single-turn numerical examples assigned one third of the total loss weight. Without a comparison that removes this group, improved numerical performance could reflect direct supervision rather than help from other tasks.

  2. The cyclic-shift comparison changes both turn order and the history available for each answer, as Appendix D.5 acknowledges. Some questions may lose earlier observations needed to answer them. This supports preserving valid histories, but does not establish an advantage over separate training examples that retain the same complete histories. Proposition 3(ii) and Appendix D.2 also state that, when each transition keeps its causal prefix and the weights are unchanged, the empirical objective is invariant to grouping turns into episodes or presenting them as separate examples, whereas the prose suggests that keeping episodes intact is itself beneficial.

  3. The training-data construction is still insufficiently specified. Appendix C.1 gives the counts and observation variants, but does not identify the specific sources of the 9,000 categorical question-answer pairs and 3,635 additional examples, or fully explain the geometry corrections for displacement and object-distance data. This makes the training set difficult to reconstruct from the manuscript alone; the release of source manifests promised in the Reproducibility Statement would help, but the sources should be named in the paper itself.

Questions for authors

Five questions are listed below. These questions may influence my understanding of the research method and the experiment results.

  1. How do the three benchmark scores change when the 2,545 single-turn numerical examples are removed, while matching exposure to the remaining examples and the total training budget as closely as possible?

  2. How does training on intact episodes compare with training on separate questions that retain exactly the same complete histories and loss weights?

  3. The introduction says existing benchmarks cover all three query types, but also says whether one model can handle them is untested. Is the intended gap multi-task evaluation, or interactions between tasks during joint training?

  4. Could you provide the categorical-data sources, the construction rules for the three observation variants, and the geometry-correction procedure, and clarify whether the 243 overlapping ScanNet scenes were removed from training or evaluation?

  5. Are both native answers and JSON-wrapped answers subject to the eight-token generation limit on UCS? Does their performance comparison remain the same when this limit is increased?

2 comments
AuthorsOpenReview ↗

A kind reminder of the upcoming end of the discussion period

Dear Reviewer 6xeB,

Thank you for your five questions. Our November 9 response reports the requested control masking the 2,545 single-turn numerical examples and the UCS comparison at common 32- and 64-token generation limits. We also clarify the data sources, label correction, and what the chronological-order experiment establishes.

We would appreciate your feedback on whether the numerical-supervision control and the equal-budget UCS evaluation address your main concerns.

Dear Reviewer 6xeB,

We have uploaded the revised manuscript with the controls and protocol details you requested. Table 21 and Appendix J.1 (p. 39) compare the original numerical labels, corrected labels, and masked single-turn numerical loss. The text now specifies what the masking experiment measures. The remaining examples retain their loss coefficients, total loss mass falls from 1 to 2/3, and the multi-turn pool still supplies numerical supervision. This qualifies how we attribute the change in estimation performance.

Appendix C.1 (p. 20) identifies the categorical data source, reports the frame budgets for all three observation variants, and corrects the description of the VLM-3R labels used in the headline run. Appendix C.3 (p. 21) specifies that the 243 overlapping ScanNet scenes were removed from the OST evaluation candidate set using training-source scene IDs.

For your decoding question, Appendix J.2 (pp. 39–40) records both the original limits and the equal-budget check. On all 8,114 UCS questions, each fixed seed-42 checkpoint generated identical token sequences at the 32- and 64-token ceilings. Every response parsed and finished below both limits. The NATIVE advantage was 3.11 percentage points at either ceiling.

The introduction (p. 1) now frames the research question around the interaction of supervision types during joint training. Section 5.3 (p. 8) also makes explicit that the cyclic shift tests the effect of changing the available history. A direct training comparison of episode grouping with identical histories remains open. Thank you for helping us make these distinctions clear.

Reviewer_3HUUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ChronoReplay, a post-training recipe for streaming spatial reasoning that combines native-format supervision, chronological multi-turn episodes, and family-balanced learning. Starting from Qwen3-VL-8B, it reports gains of 4.91, 13.41, and 7.44 points on OVO-S, OST, and UCS under matched observation access. The paper frames these results as evidence that a single checkpoint can serve categorical, metric, and temporal queries, and that preserving heterogeneous supervision with valid temporal context, together with family-balanced learning, mitigates the specialization it calls interface lock-in. My assessment focuses on whether the controls establish an interface-specific phenomenon and isolate the recipe's contribution beyond exposure to complementary supervision.

Critical strengths
  1. Consistent improvements across spatial readouts under matched observation access. Starting from Qwen3-VL-8B-Instruct, ChronoReplay improves OVO-S, OST, and UCS by 4.91, 13.41, and 7.44 points, respectively, with improvements across all fifteen OST subtypes. Although the headline scores come from one seed, the three full-objective runs in Table 10 support the direction and approximate magnitude of these gains. Table 5 also separates the training effect from the additional adaptive-access effect.

  2. Same-data controls establish benefits beyond joint data exposure. With joint data, chronological prefixes, native targets, and training budget held fixed, ChronoReplay outperforms both ordinary token SFT and natural turn weighting. The former comparison supports the combined weighting design; the latter more specifically supports explicitly setting the relative family mass, since natural turn weighting already retains answer-length normalization and within-family weighting. This advantage holds across all three benchmarks in each of three paired seeds (Tables 9–10).

  3. A simple recipe with appropriately qualified conclusions. The method requires no new memory mechanism. The paper explicitly acknowledges the crossed OST/UCS effects of native versus canonical serialization, the attribution limits of the chronological-alignment control, and the source and task-coverage differences between the single-family variants.

Critical weaknesses
  1. The empirical evidence does not isolate the role of answer-interface granularity in the observed specialization. Section 3 motivates the method through the claim that “interfaces refine one another”: categorical supervision distinguishes coarse spatial regions, whereas metric supervision resolves variation within those regions. However, the single-family comparisons also change data sources, supervised tasks, and interaction structure, so they do not isolate this proposed distinction (Appendices C.1 and D.7). The NATIVE/CANONICAL comparison usefully demonstrates that lossless output serialization can redistribute transfer across OST and UCS, but addresses a separate question from the granularity of the supervised spatial information (Appendix D.4). Thus, the results establish uneven cross-readout transfer and sensitivity to serialization, while leaving the interface-granularity explanation empirically unresolved.

  2. The answer-format comparison lacks replication across training seeds. Appendix D.4 reports one matched NATIVE/CANONICAL pair: NATIVE improves UCS by 2.92 points but lowers OST by 1.48 points relative to CANONICAL. Additional paired seeds would strengthen the evidence that this crossed transfer pattern is reproducible. The multi-seed objective comparison and the training-topology replication do not establish the robustness of the serialization intervention itself.

Questions for authors
  1. Please report the actual optimizer-update counts and total supervised assistant-token exposure for the categorical-only, trajectory-only, and full ChronoReplay runs in Table 14. Appendix D.1 specifies one epoch and a shared global batch size, but the pools differ in size and interaction structure. Were any exposure-matching adjustments made, and how should any remaining differences qualify the interpretation of the single-family comparison?

  2. Appendix D.4 reports one matched NATIVE/CANONICAL pair, with opposing effects on OST and UCS. Has this comparison been repeated across training seeds? If additional runs exist, please provide the paired differences; otherwise, please confirm that the observed serialization tradeoff is based on a single seed pair and qualify its robustness accordingly.

3 comments
AuthorsOpenReview ↗

A kind reminder of the upcoming end of the discussion period

Dear Reviewer 3HUU,

Thank you for your careful assessment of the matched-objective evidence and the interpretation of the serialization results. We would appreciate your feedback on the two additions in our November 9 response: the independently trained seed-73 pair reproducing the crossed OST/UCS pattern, and the reconstructed update and supervised-token budgets with their implications for Table 14.

Could you please let us know whether these additions address your questions about replication and training exposure? We would welcome any further clarification you would find useful.

Thank you again for your time and constructive feedback.

Dear Reviewer 3HUU,

We have uploaded the revised manuscript. Table 22 (p. 39) now reports both independently trained NATIVE/CANONICAL pairs. For the new seed-73 pair, CANONICAL scores 1.78 points higher on OST and NATIVE scores 3.24 points higher on UCS, reproducing both directions observed at seed 42. The table identifies the seed-42 UCS entry as a fresh evaluation of the original checkpoints. Appendix J.2 (p. 40) explains that the confidence intervals quantify test-source uncertainty for each trained pair.

Your exposure question also clarifies how to interpret Table 14 (p. 25). As detailed in our November 9 response, the declared schedule gives categorical-only training 958 updates and 61,280 supervised tokens, compared with 200 updates and 194,662 tokens for trajectory-only training. These single-family figures are reconstructions from the eight-GPU schedule. For the new full reference run, the execution trace directly records 1,237 updates and 267,211 supervised tokens (p. 39).

The categorical-only pool therefore has more updates, while the trajectory-only pool supplies more supervised tokens. We interpret Table 14 as a comparison of specialization across those training pools. Section 3 (p. 3) explicitly identifies information granularity as an open explanation, and §5.3 (p. 8) presents the replicated answer-format tradeoff separately. Thank you for drawing attention to both distinctions.

Thank you for the additional seed-73 pair and the exposure audit. These resolve the two procedural questions I raised, and I do not require further experiments. On the original weakness: we now agree that the single-family comparison does not isolate the information-granularity hypothesis, and that this explanation remains unresolved. I accept that leaving this as a stated limitation. My assessment is unchanged.

Reviewer_RjsjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • proposes a SFT recipe for Qwen3-VL-8B on streaming spatial QAs, "ChronoReplay"
    • data sources:
      • single-turn: ~12.6k MCQAs + ~2.5k numerical QAs from ARKitScenes and other unnamed sources
      • multi-turn: ~6k training conversations from OST-Bench
    • loss is weighted as 2/3 for single-turn, and 1/3 for multi-turn
    • improves Qwen-VL mean performance on the three test sets (48.28 -> 56.86). an "ordinary token SFT" baseline on the same mix improves to 54.10
  • also proposes a frame selection rule that improves performance at inference time
  • the recipe is framed as addressing "interface lock-in" through "temporal-origin alignment"
Critical strengths
  1. one checkpoint improves all three benchmarks over the same base model
Critical weaknesses
  1. the presentation is so poor it is very difficult to understand what the paper does. it took me several passes of the paper before I could understand what the paper was doing
  2. the paper is a simple fine-tuning recipe, papered heavily with fancy terminology and jargon. it is filled with conceptual overclaiming, especially for such a simple implementation.
    • "native target format" simply means that the raw dataset string outputs are kept without applying any translation / preprocessing. this is the simplest default (do nothing).
      • $\S$1 motivates this against "collapsing it into one canonical answer type", but the only control is a lossless json wrapper (doesn't collapse anything) so this is never validated
    • "preservation" vs "trajectory" family is fundamentally just single-turn vs multi-turn, dressed up in theoretical language
    • "temporal-origin alignment" simply means that multi-turn conversation rounds are fed in in their natural chronological order, instead of eg shuffling the conversation (or "cyclically rotating") turns (and thus breaking temporal causality...)
      • furthermore, the ablations of this (Tab. 12) only show that temporally corrupted data is worse than the correct normal order
  3. most of the gain comes from just SFT'ing on their data mix (Tab. 9), which is not even a novel contribution in itself. the supposedly novel parts add 1-3 points
Questions for authors

how is this paper anything more than a simple standard data recipe?

3 comments
AuthorsOpenReview ↗

A kind reminder of the upcoming end of the discussion period

Dear Reviewer Rjsj,

We would appreciate your feedback on our November 9 response. It highlights the same-data objective controls, including positive family-allocation gains in all nine seed–benchmark comparisons, and an additional training pair reproducing the OST/UCS serialization tradeoff. We also clarified the scope of the chronology result and the distinction between serialization sensitivity and the information-granularity hypothesis.

Could you please share whether these results and clarifications address your concerns about the empirical contribution and soundness? We would appreciate your updated assessment and any specific remaining concerns that we can address during the discussion.

Thank you for your time and consideration.

thank you for the response. I will keep my rating at 1

the rebuttal largely concedes my main critiques rather than addressing them:

  • native targets are "a straightforward construction choice"
  • the granularity motivation "remains a hypothesis"
  • chronology ablations only show corrupted order is worse; any benefit of episode grouping "remains experimentally open"
  • joint SFT supplies most of the gain (Tab. 9); the method adds +2.76 over plain SFT, family weighting ~1 pt (Tab. 10)

furthermore, every fix is a promise and no revision was uploaded, so the presentation and overclaiming issues still stand. with the framing corrected, what remains is a simple data mix + loss-weighting recipe, not enough for ICLR

Dear Reviewer Rjsj,

Thank you for explaining your assessment. When you posted your follow-up, our response described proposed changes and the original PDF was still on record. We have now uploaded the revised manuscript.

The abstract and introduction (pp. 1–2) state what the experiments establish beyond joint data exposure. With the same training records, explicitly balancing the supervision weights improves all three benchmarks over natural turn weighting in each of three paired runs. Changing the answer format produces a different outcome: across two independently trained pairs, JSON favors OST and source-native answers favor UCS. Figure 1 (p. 2) now labels the model “One jointly trained checkpoint” and shows the tasks on which we evaluate it. Section 3 (p. 3) identifies information granularity as a possible explanation that remains to be tested.

We also revised §5.3 (p. 8) to address your point about chronology directly. A cyclic shift changes the history available to each answer, so the observed OST advantage supports retaining valid histories. The benefit of grouping turns when their histories are identical remains open. The same page now discusses a concrete limit on transfer: changing the auxiliary supervision raises held-out relation accuracy from 62.29 to 66.88 while the mean on the three native benchmarks falls from 56.42 to 56.06.

We appreciate your insistence that readers should be able to see these distinctions in the paper itself.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response on supervision weighting, answer format, and spatial transfer

We thank the reviewers for asking what ChronoReplay establishes beyond training on a mixture of spatial tasks. Their questions led us to revisit the phrase “learns reusable spatial state” in our TL;DR. Our experiments measure performance on spatial tasks, and we propose the following replacement:

With training data and observation access held fixed, balancing supervision across categorical, numerical, and trajectory examples improves all three spatial benchmarks. A separate comparison shows that changing the answer format shifts performance between OST and UCS, even when both formats encode the same answer.

What does weighting add when the training examples are identical? Reviewer Rjsj correctly observes that ordinary joint SFT accounts for much of the improvement. Table 9 quantifies both contributions. The three-benchmark mean rises from 48.28 for the base model to 54.10 for ordinary token SFT, then to 56.86 for ChronoReplay on the same records, with the same histories and observation access. Table 10 asks a narrower question. Both objectives already normalize loss within answers and retain the same within-family weighting. Setting each family’s total loss weight explicitly improves OVO-S by 0.92 ± 0.27 points, OST by 1.43 ± 0.19, and UCS by 1.17 ± 0.25. These are means and sample standard deviations of the differences across three paired training seeds. Each benchmark improves in every pair.

Does better performance on an auxiliary spatial task predict better native answers? Appendix H, Table 19 gives a direct test. With four auxiliary relations per state in both conditions, switching from same-type to cross-type supervision raises accuracy on held-out relation judgments from 62.29% to 66.88%. Over the same comparison, the mean on the three native benchmarks falls from 56.42 to 56.06. This result explains why we evaluate transfer on the native tasks themselves. It also places a concrete limit on our interpretation of “reusable spatial state.” The granularity of the information learned by the model remains an open question.

What did the additional serialization experiment establish? Reviewer 3HUU asked whether the OST/UCS tradeoff would recur after independent training. With seed 73, CANONICAL scores 1.78 points higher on OST and NATIVE scores 3.24 points higher on UCS, reproducing both directions from seed 42. Reviewer 6xeB raised a different possible explanation for the UCS result: a short generation budget could disadvantage JSON answers. For each fixed seed-42 checkpoint, the generated tokens were identical at ceilings of 32 and 64 tokens on all 8,114 UCS questions. Every answer parsed and finished below both ceilings. The independent training pair addresses replication, and the decoding check addresses truncation. The format comparison holds answer information, training records, and optimizer updates fixed. Supervised-token counts vary with the encoding.

We will use the same precision in describing chronology. Chronological order outperforms cyclic shifts by 2.42 and 2.41 OST points in two paired runs. Shifting changes the history available to each answer, so this comparison supports preserving valid histories. When separately presented questions retain the same full histories, targets, and loss coefficients, Proposition 3(ii) gives the same empirical objective. Whether episode grouping changes optimization in that setting remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.