acceptodds
Under review as a conference paper at ICLR 2027

CaRAMeL: Compressed and Retrieval-Augmented Spatio-Temporal Memory for Long-Horizon Robot Navigation

Abstract

Caption-based spatio-temporal memory lets a robot answer where and when something happened over hours of operation, but two costs keep it off the robot: the captioner's visual-token budget and a reasoning LLM that such systems assume to be a frontier API model. Consecutive frames from a moving camera repeat the same scene, consecutive captions repeat whenever the robot is stationary or revisits a place, and the quantitative answers the memory is queried for are anchored to the pose and timestamp stored with each entry rather than to its caption; a local reasoner declines to transcribe those fields on close to half of the quantitative questions, so its dependence on an API model is one of coverage rather than accuracy. CaRAMeL acts on each observation without training. Evidence-grounded decoding grounds quantitative fields in the retrieved metadata. Chimera token compression transplants the group-of-pictures structure of video codecs into the vision encoder, keeping one reference frame per group intact and carrying only the changed regions in a second frame, so that no token pools across observations. Caption compression merges consecutive captions only while the robot's odometry keeps it within a bounded radius, and every segment's pose and timestamp remain in the store. On NaVQA, an 8B local reasoner on two-thirds of the visual tokens and a 40% smaller store reaches the positional and temporal error of the same memory driven by a frontier API model (33.7 against 34.4 m), with the lowest hallucination rate among compression methods and a 1.2x faster captioner on a Jetson AGX Orin.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot navigation, spatio-temporal memory, visual token compression, vision-language models, on-device inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9vebOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper shows two deployment costs of caption-based spatio-temporal memory for robot navigation: visual-token overhead during captioning and reliance on frontier API models for querying.

To address them, the authors propose CaRAMeL, a training-free pipeline with three key stages: (1) Chimera token compression, (2) physics-gated caption compression, and (3) evidence-grounded decoding, to address

On NaVQA, CaRAMeL makes a local Qwen3-8B model match GPT-5-mini on spatial and temporal error using 33% fewer visual tokens and 40% fewer memory entries, while running 1.22x faster on a Jetson AGX Orin.

Critical strengths
  1. Clear diagnostic analysis of navigation memory: The ablations prove that NaVQA's spatial and temporal metrics test metadata transcription after retrieval, masking caption corruption. Evaluating compression directly on person hallucinations and text collapse exposes a major flaw in cross-frame token pooling.
  2. Simple, effective token and caption compression: Preserving single-frame token provenance via 4-connected components avoids pooling artifacts and reduces Qwen3-8B long-horizon spatial error. The odometry gate prevents merging captions across different rooms, cutting ~40% of entries without hurting accuracy.
Critical weaknesses
  1. Underspecified evidence-grounded decoding: The paper states that quantitative fields are read from the retrieved entry whose caption best matches the question and replace the LLM's output when it is absent or disagrees. However, it never explains how the best-matching entry among the k=4k=4 retrieved slots is selected or what threshold defines disagreement. If a heuristic always overwrites the LLM's numbers, the reasoner is bypassed after retrieval, which needs clarification for reproducibility.
  2. Non-causal per-sequence threshold calibration: The Otsu velocity threshold, 90th-percentile stationary radius bounds, and similarity quantiles are calibrated per sequence. In live deployment, full-sequence statistics are unavailable in advance; these should be fixed across sequences or estimated online.
Questions for authors
  1. In Section 3.4, how does evidence-grounded decoding select the "best matching" entry, or set of entries for duration, from the top-k retrieved context, and how much of the gain in Table 2a comes from resolving refusals versus overwriting non-refusal LLM numbers?

  2. How does caption compression perform if the velocity, radius, and similarity thresholds are held fixed across sequences (or calibrated online from a short warmup) instead of computed per sequence?

  3. What is the full per-segment latency on the Orin when autoregressive caption decoding is included, and can the 13B captioner and 8B reasoner co-reside in memory?

3 comments

We sincerely thank the reviewer for the positive assessment and the precise questions, each of which led to a correction or a new experiment. We will upload a revised PDF shortly, with changes marked in blue.

W1 and Q1 (EG specification). Our §3.4 description was imprecise, and we will add Algorithm 1 with the exact rules. EG picks the top-k entry with the largest token overlap with the question, breaking ties by retrieval rank. Positions are only filled, never overwritten; time and duration are replaced when missing or off by more than 1.0 min, a constant fixed before evaluation. To separate the two sources of gain, we logged the LLM's parsed answer and the EG candidate before replacement and re-scored the same outputs counterfactually (chimera/784; Qwen3-8B seed 1, with the same pattern in seeds 2–3):

Reasoner, fieldLLM only (refusals)Fill refusals onlyEGAlways EG's own entry
Qwen3-8B, position (m)440.0 (42.3%)37.537.538.1
Qwen3-8B, time (min)387.2 (38.5%)3.311.851.87
GPT-5-mini, position (m)50.8 (1.4%)40.440.454.8 (p = .015)
GPT-5-mini, time (min)53.0 (5.1%)1.731.942.00

Errors in the "LLM only" column include the benchmark's 1000-unit penalty for refusals. The position gain comes entirely from filling refusals, and time overwriting corrects the weaker reasoner (Qwen3-8B 3.31 → 1.85 min, p = .074). Overwriting helps most for the weakest reasoner: across three seeds of Llama-3.1-8B, it reduces the error of overwritten time answers from 6.7–12.4 min to 2.6–2.8 min. Always using EG's entry worsens GPT-5-mini by 14.4 m, so the reasoner is not bypassed: it selects the memory, and EG transcribes it.

W2 and Q2 (non-causal calibration). Checking this revealed that Appendix A.6 misdescribed the implementation: similarity thresholds come from the first 20 caption pairs and radius bounds are fixed, so only the Otsu speed threshold is non-causal. Replacing it as suggested:

Calibration (3 seeds)Entry reductionSpans > 4 mNaVQA position / temporal, answered items
As submitted39.5%035.5 m / 2.03 min
Leave-one-sequence-out38.7%032.9 m / 2.12 min
Fixed default thresholds54.6%038.4 m / 2.09 min
Warm-up, first 2 min34.6%032.6 m / 1.96 min
Warm-up, first 5 min33.3%035.7 m / 2.21 min
Streaming update28.8%034.7 m / 2.25 min

Refusals are at most 0.9% in every row. No variant, including thresholds held fixed across sequences, differs significantly from the submitted calibration in position, time, or binary accuracy (paired tests including the refusal penalty, all p > .2). We will correct §3.3 and Appendix A.6 and add this table to the appendix.

Q3 (full latency and co-residence). Our §4.6 timing omitted caption generation, which we will correct. Generation dominates per-segment time, so our target is throughput under the memory limit, and both models co-reside:

Orin, generation includedUncompressedChimera 784Chimera 392
Encode + prefill, batch 1 (s)3.091.591.50
Total per segment, batch 1 (s)26.725.826.2
Peak throughput (segments/s)0.1210.168 (1.39×)0.221 (1.83×)
Max batch, reasoner resident141820

With Qwen3-8B (Q4) co-resident, peak memory is 49.0 of 61.4 GB while both run, and chimera keeps 1.41–1.58× the captioning throughput.

We thank the reviewer again for the supportive review, which helped us correct the paper and sharpen its claims. We would be glad to answer any further questions.

Thank you for the response, additional experiments and revision.

My concerns are resolved and I will maintain my score.

Dear Reviewer,

Thank you for taking the time to read our response and the revised paper, and for confirming that your concerns are resolved. Your questions on the decoding specification, the calibration, and the on-device measurement led directly to corrections and new experiments that made the paper better. We appreciate your careful review.

Best regards,

The Authors

Reviewer_WqdsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for improving the efficiency of memory construction, storage, and retrieval for long-horizon robot navigation. Building upon existing caption-based spatio-temporal memory methods, particularly ReMEmbR, the proposed approach introduces three main techniques: chimera token compression, physics-gated caption compression, and evidence-grounded structured decoding. The effectiveness of the proposed framework is evaluated primarily on the NaVQA benchmark, with comparisons against existing memory-based approaches and token compression methods. The experimental results demonstrate improvements in computational efficiency and memory storage, while generally maintaining competitive question-answering performance. The authors also provide experiments on an embedded hardware platform to evaluate the efficiency of memory construction.

Critical strengths
  • The paper addresses meaningful computational and memory constraints in long-horizon robotic navigation, particularly for systems operating with limited onboard computational resources.
  • The proposed methods are straightforward and do not require additional training.
  • The experiments demonstrate reductions in visual-token consumption and memory storage, along with improvements in captioning efficiency on an embedded platform.
Critical weaknesses
  • My primary concern is the relatively limited novelty of the proposed framework. While the overall approach is intuitive and practically motivated, the three main components largely build upon established concepts. The overall contributions appear to be incremental improvements to an existing memory-based navigation pipeline rather than fundamentally new methods.
  • The main experiments are conducted on a single benchmark NaVQA. While the results demonstrate the effectiveness of the proposed framework in this setting, it remains unclear whether similar improvements can be achieved across more diverse environments and tasks.
Questions for authors
  • Could the authors clarify how much of the overall performance improvement can be attributed to key techniques adopted, including memory compression and evidence-grounded decoding?
2 comments

We thank the reviewer for the time and constructive feedback, and address the three points below with new experiments. We will upload a revised PDF shortly, with changes marked in blue.

W1 (limited novelty). The components are deliberately simple and training-free, since memory construction runs on the robot's own budget. Our contribution lies in a finding about caption memory and what it implies, which the new experiments test directly. (1) Quantitative answers are transcriptions of stored odometry, and small reasoners fail by refusing: in 40% of audited refusals, the correct entry was in context. (2) Grounding is needed, and prompting does not replace it:

Qwen3-8B, chimera/784, 3 seedsPosition refusalsPosition, median (m)Temporal (min)
Free-form45.1%97.3351.9
+ explicit "do not refuse" instruction37.1%51.6275.3
JSON-schema decoding0%23.53.49
EG0.5%16.22.03

Errors include the benchmark's 1000-unit penalty for refusals. EG also transfers to an unmodified ReMEmbR pipeline (Qwen3-8B: 462 → 38 m). (3) Merging tokens corrupts captions while selecting them does not (hallucination 12.0–19.6% vs 3.8–5.1%, Table 3 of the revision), which QA metrics cannot detect because quantitative answers bypass the captions. We will revise the introduction to state these three contributions explicitly.

W2 (single benchmark). We added OC-NaVQA, which changes the task from the robot's position to object positions and uses full-sequence memory of up to 35.8 min:

OC-NaVQA, GPT-5-miniPosition error (m)Binary acc.
ReMEmbR + NVILA-Lite-2B / 8B (published)53.5 / 55.9.432 / .463
DAAAM, 3D scene graph (published)41.75.711
Ours, chimera/784 + EG45.2.642

The seven sequences cover indoor corridors and outdoor walkways, and the main findings are reported across two captioners and five reasoners. Extending the evaluation to further environments is a natural next step. Our memory approaches DAAAM while being far cheaper to build (a fresh run of DAAAM's public pipeline on the same GPU):

One 13.7-min sequence, same GPU (RTX PRO 6000)Ours (captioning + compression)DAAAM (public pipeline)
Build time14.1 min (1.03× sequence length)54.1 min (3.96× sequence length)
Energy (GPU)44.9 Wh90.7 Wh
Host memory, peak6.1 GiB50.8 GiB
GPU memory, peak34.7 GiB (VILA1.5-13B)28.4 GiB (DAM-3B)

Q1 (contribution of each component). We ran a full 2×2×2 ablation (chimera × caption compression × EG; Qwen3-8B, 3 seeds):

ComponentEffect on accuracyEffect on cost
EGPosition −436.7 m (p < 10⁻¹¹), almost all from removing refusals (−45.5 points)—
ChimeraNot significant33–67% fewer visual tokens
Caption compressionNot significant~40% fewer stored entries
LLM (on OC-NaVQA)Position 78.1 → 45.2 m when added (p = .013)—

EG also transfers to an unmodified ReMEmbR pipeline:

ReMEmbR + EGPosition (m), EG off → onRefusals, off → on
Qwen3-8B462.0 → 38.144% → 0%
Llama-3.1-8B310.7 → 173.726% → 11%
GPT-5-mini45.8 → 34.11.4% → 0%

Both tables will be added to the appendix.

We thank the reviewer again; the feedback led us to test each component separately and to evaluate beyond NaVQA. We would be glad to answer any further questions.

Dear Reviewer,

We appreciate the time you spent on our paper. Following your comments, we added a second benchmark (OC-NaVQA) and an ablation that separates the contribution of each component; both are in our response and in the revised PDF. We would be glad to know whether these additions answer your questions, and we are happy to clarify any remaining point while the discussion is open.

Best regards,

The Authors

Reviewer_tYNyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses two main bottlenecks in deploying caption-based spatio-temporal memory on a robot: the high compute cost of captioning video segments with a vision-language model, and the reliance on cloud API models to answer quantitative where and when questions. To resolve these issues without training, the authors propose three modifications to the pipeline. First, Chimera token compression reduces the visual token budget inside the vision encoder by keeping one reference frame intact within each temporal group and constructing a second "chimera" frame from the 70% of patch positions that changed the most, filling each 4-connected spatial region from a single member frame. Second, physics-gated caption compression prunes, deduplicates, or merges consecutive captions based on scene and object text similarity, but only while odometry indicates the robot remains within a bounded radius and does not cross a stop-to-move transition. Finally, evidence-grounded decoding extracts quantitative answers directly from the pose and timestamp metadata of the retrieved entries at query time, overwriting the language model's generated number whenever the model refuses to answer or disagrees.

Critical strengths

The authors show that small local LLMs fail on NaVQA primarily because they refuse to output numbers on almost half of the quantitative questions, which triggers a benchmark penalty, and that quantitative answers depend on the prompt's odometry headers rather than the caption text.

Instead of relying only on downstream QA scores, the paper checks the captions directly and shows that cross-frame token pooling causes 13B VLMs to hallucinate people and fall into repetitive text loops, whereas keeping tokens unpooled avoids this.

Critical weaknesses
  1. Whenever the LLM's answer is missing or disagrees with the retrieved metadata, the system overwrites it. Positions are taken from a single matching entry, while time and duration answers are computed from stored timestamps. This leaves unclear whether the system can handle more compositional queries, such as the distance between two sightings or comparisons across multiple observations. It works on NaVQA largely because the benchmark grades the robot's position when it saw an object, not the object's own position. Furthermore, the quantitative evaluation contains only 71 spatial and 39 temporal questions, making it difficult to establish how robust the apparent parity with GPT-5-mini is beyond the particular question types in NaVQA.
  2. Chimera is not shown to beat uniform frame subsampling. At the main token budget, uniform subsampling gives the same token reduction with a simpler compression procedure, and its hallucination rate is close to Chimera's. Neither shows a statistically significant increase in person hallucination relative to uncompressed captioning, yet uniform subsampling is left out of the main NaVQA table. At the default 70% coded fraction, I suspect that the largest connected component may dominate the coded grid, causing much of the Chimera slot to come from a single member frame rather than combining distinct observations.
  3. Caption compression is calibrated offline and ignores revisits. The speed threshold, radius bound, and similarity thresholds are computed over each full sequence, which a robot streaming new data cannot do without an online calibration mechanism. Only consecutive segments are merged, so revisits to the same location remain separate entries. As the authors acknowledge, the store still grows linearly with mission duration, limiting the method's usefulness for long-term memory scalability. Merged spans retain individual timestamps, but retrieval represents them using the span-start timestamp. This can misrepresent when an object first appeared during a long stationary span, even though its original timestamp remains stored.
  4. All experiments are offline replays of recorded NaVQA sequences, each under half an hour long. The Jetson timing covers only vision encoding and prefill for an unquantized 13B captioner and leaves out the time to generate the caption text. The 8B reasoner never runs on the Jetson. The smaller or quantized captioners usually deployed on edge robots are not tested with the compression pipeline.
Questions for authors
  1. Evidence-grounded decoding takes quantitative fields directly from the retrieved metadata. How does a simple baseline without an LLM perform on the quantitative NaVQA categories, where the question is embedded, the top entry is retrieved, and its pose or timestamp is returned directly?
  2. How does uniform frame subsampling at the main token budget perform on NaVQA? At the default coded fraction, how much of the coded set falls in the largest connected region, and how often does a Chimera slot take nearly all its coded tokens from one member frame?
  3. What is the full per-segment captioning time on the Jetson once caption text generation is included? What are the memory requirements for running the 13B captioner and 8B reasoner on the same device, and is concurrent execution feasible?
  4. Free-form Qwen3-8B answers only about half the quantitative questions, while GPT-5-mini answers nearly all of them. What are the spatial and temporal errors on just the questions both models answered?
2 comments

Thank you for the careful and constructive review. Every question led to a new experiment; full tables will follow in the revised PDF (changes in blue).

W1. (a) Overwriting. §3.4 was imprecise: EG fills a position only when the LLM gives none and never overwrites it; time and duration are replaced only when missing or off by more than 1.0 min (fixed in advance). Counterfactual re-scoring shows the position gain comes entirely from filling refusals, and forcing EG's own entry worsens GPT-5-mini by 14.4 m (p = .015), so the LLM still selects the memory. (b) Compositional queries go to the LLM with poses and timestamps serialized, as in prior caption memory; neither benchmark contains them, so evaluating them is a natural next step. (c) Robot vs. object position. We added OC-NaVQA (object positions, full-sequence memory):

OC-NaVQA, GPT-5-miniPosition (m)BinaryBuild time*Energy*
ReMEmbR (published)53.5–55.9.432–.463——
DAAAM (published)41.75.71154.1 min90.7 Wh
Ours45.2.64214.1 min44.9 Wh
Ours, no LLM78.1———

*Measured by us for one 13.7-min sequence on the same GPU (DAAAM via its public pipeline): ours builds 3.8× faster at half the energy. Host / GPU memory: ours 6.1 / 34.7 GiB, DAAAM 50.8 / 28.4 GiB. (d) Robustness. Over 3 seeds, Qwen3-8B + EG (40.1 ± 4.7 m, 2.03 min) is statistically indistinguishable from GPT-5-mini + EG (37.4 m, 1.79 min over 2 runs; p = .64, .23), a gap smaller than GPT-5-mini's own run-to-run difference (6.0 m).

W2 / Q2. (a) Uniform sampling. Mixed at 784 tokens; chimera better preserves transient content at 392:

Metric (token budget)UniformChimerap
Time-of-event error, Qwen3-8B (784)2.03 min1.35 min.066
Time-of-event error, GPT-5-mini (784)1.61 min0.95 min.032
Position error, GPT-5-mini (784)25.0 m40.4 m.014
Collapsed captions (784)30.25
Transient person mentioned (392, pre-registered)31.5%41.9%.018

Selection keeps captions clean while merging corrupts them (hallucination: chimera 3.8%, uniform 5.1%, VisionZip 5.1% vs AOT 12.0%, ToMe 19.6%; Table 3 of the revision). (b) Largest component. It covers 90.8% of the coded set, and one member supplies ≥90% of coded tokens in 94.7% of slots (G = 2). This follows geometrically from coding 70% of the grid: random masks of the same size give 94.8%, whereas at ρ = 0.1 our regions are far more coherent than random (36.7% vs 5.1%). Each region is filled from one instant by design; per-position mixing caused 6 collapsed captions and +36 m LONG error.

W3. (a) Calibration. Appendix A.6 misdescribed the implementation: only the Otsu speed threshold uses the full sequence. With leave-one-sequence-out, fixed, warm-up, or streaming calibration (3 seeds), entry reduction stays at 29–55%, no span exceeds 4 m, and accuracy is unchanged (all p > .2). (b) Growth. Keeping every pose and timestamp costs only 3.6 MB per hour after compression; revisits stay separate to preserve their timestamps and can be grouped at retrieval time. (c) Span start. Merged spans average 5.8 s, and the span-start offset accounts for 0.18% of the time error.

W4 / Q3. (a) Offline replays. A fair point: both benchmarks are recordings, and we did not run on a robot. The pipeline is built for online use, though: compression is causal per segment, calibration can be streamed (W3a), and timing is measured on the target device. OC-NaVQA already uses full sequences of up to 35.8 min, and a live, longer deployment is our next step. (b) §4.6 omitted generation, and we apologize. With generation, a segment takes about 26 s at batch size 1 in every setting, as decoding dominates for any visual-token method; chimera halves encode + prefill (3.09 → 1.59 s). Our target is throughput under the memory limit:

Orin, generation includedMax batchPeak segments/s
Uncompressed180.121
Chimera 784210.168 (1.39×)
Chimera 392280.221 (1.83×)

(c) Captioner and 8B reasoner co-reside and run concurrently (peak 49.0 of 61.4 GB); with the reasoner resident, chimera keeps 1.41–1.58× throughput. (d) Smaller and quantized captioners (batch size 1):

CaptionerUncompressedChimeraMemoryHallucination
VILA1.5-13B, FP1626.7 s25.8 s30.3 GB—
VILA1.5-13B, 4-bit18.8 s17.6 s14.0 GBunchanged (p = .69)
NVILA-Lite-8B22.7 s19.8 s19.7 GBsame as uncompressed (4.4%)

Q1. Retrieval + EG with no LLM reaches 31.9 m / 1.64 min on NaVQA, indistinguishable from both agents (p = .13, .57), but 78.1 m on OC-NaVQA against 45.2 m with GPT-5-mini (p = .013): the LLM contributes retrieval for object queries, and EG transcription.

Q4. On items both models answered (free-form, 4 runs), Qwen3-8B vs GPT-5-mini is 35.1 vs 32.5 m and 3.09 vs 2.10 min, not significant in 3 of 4 runs; the small model's deficit is coverage (45% vs 4% refusals), which EG closes.

Dear Reviewer

Thank you again for your time and feedback. Before the discussion closes, we wanted to check whether our response and the revised PDF address your concerns. If anything remains unclear, we would be glad to clarify.

Best regards, The Authors

Comments on the paper

1 comment

We thank all reviewers for the time and care they put into reading our paper. The revised manuscript is now uploaded: every change is marked in blue, and the main text fits in 10 pages. The reviews pointed to four areas, and the revision covers each of them.

  • What the benchmark measures and what each component contributes: a baseline without an LLM, results on OC-NaVQA with object-position ground truth, a component ablation, and a decomposition of evidence-grounded decoding with prompting controls (§4.2–§4.4, Appendix A.17, A.18, A.22).
  • Robustness: 3-seed results for the main comparison, causal calibration of caption compression, and an exact specification of the decoder as Algorithm 1 (Table 1, §3.3, §3.4, Appendix A.16, A.17).
  • Token compression: comparisons with uniform frame sampling, ToMe and VisionZip, component statistics of the coded set, and a pre-registered transient-event test (Tables 1 and 3, Appendix A.15, A.20).
  • Practical efficiency: end-to-end on-device measurements including caption generation, throughput under the memory limit, co-residence with the reasoner, quantized and smaller captioners, and construction cost against DAAAM (§4.6, Figure 2, Appendix A.12, A.19).

Descriptive accuracy of single-run rows is now computed over the benchmark's fixed set of 67 binary questions throughout. The numbers behind each item are in our responses to the individual reviews. The reviews made this a more careful paper, and we will answer any follow-up questions promptly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.