- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper shows two deployment costs of caption-based spatio-temporal memory for robot navigation: visual-token overhead during captioning and reliance on frontier API models for querying.
To address them, the authors propose CaRAMeL, a training-free pipeline with three key stages: (1) Chimera token compression, (2) physics-gated caption compression, and (3) evidence-grounded decoding, to address
On NaVQA, CaRAMeL makes a local Qwen3-8B model match GPT-5-mini on spatial and temporal error using 33% fewer visual tokens and 40% fewer memory entries, while running 1.22x faster on a Jetson AGX Orin.
Critical strengths
- Clear diagnostic analysis of navigation memory: The ablations prove that NaVQA's spatial and temporal metrics test metadata transcription after retrieval, masking caption corruption. Evaluating compression directly on person hallucinations and text collapse exposes a major flaw in cross-frame token pooling.
- Simple, effective token and caption compression: Preserving single-frame token provenance via 4-connected components avoids pooling artifacts and reduces Qwen3-8B long-horizon spatial error. The odometry gate prevents merging captions across different rooms, cutting ~40% of entries without hurting accuracy.
Critical weaknesses
- Underspecified evidence-grounded decoding: The paper states that quantitative fields are read from the retrieved entry whose caption best matches the question and replace the LLM's output when it is absent or disagrees. However, it never explains how the best-matching entry among the retrieved slots is selected or what threshold defines disagreement. If a heuristic always overwrites the LLM's numbers, the reasoner is bypassed after retrieval, which needs clarification for reproducibility.
- Non-causal per-sequence threshold calibration: The Otsu velocity threshold, 90th-percentile stationary radius bounds, and similarity quantiles are calibrated per sequence. In live deployment, full-sequence statistics are unavailable in advance; these should be fixed across sequences or estimated online.
Questions for authors
-
In Section 3.4, how does evidence-grounded decoding select the "best matching" entry, or set of entries for duration, from the top-k retrieved context, and how much of the gain in Table 2a comes from resolving refusals versus overwriting non-refusal LLM numbers?
-
How does caption compression perform if the velocity, radius, and similarity thresholds are held fixed across sequences (or calibrated online from a short warmup) instead of computed per sequence?
-
What is the full per-segment latency on the Orin when autoregressive caption decoding is included, and can the 13B captioner and 8B reasoner co-reside in memory?
3 comments
Thank you for the response, additional experiments and revision.
My concerns are resolved and I will maintain my score.
Dear Reviewer,
Thank you for taking the time to read our response and the revised paper, and for confirming that your concerns are resolved. Your questions on the decoding specification, the calibration, and the on-device measurement led directly to corrections and new experiments that made the paper better. We appreciate your careful review.
Best regards,
The Authors
We sincerely thank the reviewer for the positive assessment and the precise questions, each of which led to a correction or a new experiment. We will upload a revised PDF shortly, with changes marked in blue.
W1 and Q1 (EG specification). Our §3.4 description was imprecise, and we will add Algorithm 1 with the exact rules. EG picks the top-k entry with the largest token overlap with the question, breaking ties by retrieval rank. Positions are only filled, never overwritten; time and duration are replaced when missing or off by more than 1.0 min, a constant fixed before evaluation. To separate the two sources of gain, we logged the LLM's parsed answer and the EG candidate before replacement and re-scored the same outputs counterfactually (chimera/784; Qwen3-8B seed 1, with the same pattern in seeds 2–3):
Errors in the "LLM only" column include the benchmark's 1000-unit penalty for refusals. The position gain comes entirely from filling refusals, and time overwriting corrects the weaker reasoner (Qwen3-8B 3.31 → 1.85 min, p = .074). Overwriting helps most for the weakest reasoner: across three seeds of Llama-3.1-8B, it reduces the error of overwritten time answers from 6.7–12.4 min to 2.6–2.8 min. Always using EG's entry worsens GPT-5-mini by 14.4 m, so the reasoner is not bypassed: it selects the memory, and EG transcribes it.
W2 and Q2 (non-causal calibration). Checking this revealed that Appendix A.6 misdescribed the implementation: similarity thresholds come from the first 20 caption pairs and radius bounds are fixed, so only the Otsu speed threshold is non-causal. Replacing it as suggested:
Refusals are at most 0.9% in every row. No variant, including thresholds held fixed across sequences, differs significantly from the submitted calibration in position, time, or binary accuracy (paired tests including the refusal penalty, all p > .2). We will correct §3.3 and Appendix A.6 and add this table to the appendix.
Q3 (full latency and co-residence). Our §4.6 timing omitted caption generation, which we will correct. Generation dominates per-segment time, so our target is throughput under the memory limit, and both models co-reside:
With Qwen3-8B (Q4) co-resident, peak memory is 49.0 of 61.4 GB while both run, and chimera keeps 1.41–1.58× the captioning throughput.
We thank the reviewer again for the supportive review, which helped us correct the paper and sharpen its claims. We would be glad to answer any further questions.