acceptodds
Withdrawn from ICLR 2027

Selective Recall for Late-Bound Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

People encounter a stream of events throughout the day, yet a later request—such as fetching an object put away hours earlier—may depend on only a few relevant memories. Inspired by this everyday need for selective recall, we study how robots can use accumulated experience when the task is not known in advance. Existing manipulation-memory evaluations often reveal goals during history encoding, place key evidence at fixed temporal positions, or associate each history with a single task, leaving task-conditioned access to rich, task-agnostic experience insufficiently evaluated. We introduce late-bound manipulation: a robot first experiences a long, continuous history containing numerous candidate events interleaved with background interactions, and only a subsequent instruction determines which evidence is needed for action. We develop MOSAIC, which pairs identical histories and execution-start states with different instructions requiring evidence from different temporal intervals. Its shared-history construction enables controlled comparisons of behavior under different subsequent instructions. We further propose ReWAM, a selective-memory world-action model that combines compact global memory, instruction-informed retrieval of detailed historical chunks, and recent local context for video and action prediction. On seven RMBench tasks, ReWAM achieves 50.57% mean success, compared with 55.00% for dense-history conditioning and 44.71% for dense visual history with local action context. On the evaluated three- and six-target MOSAIC configurations, it achieves 93% and 45% success, compared with 36% and 26% for dense-history conditioning. In separate timing experiments at 256 historical video chunks, ReWAM achieves a 3.06× full-cycle inference speedup over dense-history conditioning, with latency increasing by only 4.27% from 1 to 256 chunks. Together, MOSAIC and ReWAM provide a framework for studying task-conditioned recall and its performance–cost trade-offs in long-history manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, robot manipulation, embodied memory, long term memory, task conditioned retrieval, world action models, sparse attention, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UWaTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Late-Bound Robotic Manipulation and introduces MOSAIC, a benchmark, and ReWAM, a selective-memory model built upon LingBot-VA. MOSAIC uses shared interaction histories and execution-start states to evaluate how models utilize historical information under different subsequent instructions. ReWAM constructs learnable compact register representations for historical video chunks and selectively accesses detailed historical information through instruction-conditioned retrieval to reduce historical attention costs during prediction.

On RMBench, ReWAM achieves a mean success rate of 50.57%, compared with 55.00% for Dense and 44.71% for Local-A. On the three-target and six-target Store and Retrieve Blocks configurations of MOSAIC, it achieves 93% and 46% success. At 256 historical chunks, ReWAM also achieves a 3.06× model-side inference speedup over Dense.

Critical strengths

S1. Meaningful task formulation and benchmark design.

MOSAIC focuses on the practically relevant late-bound instruction setting in long-term robotic memory. Its shared-history protocol fixes the interaction history and execution-start state across different instructions, providing a more controlled setting for evaluating instruction-dependent use of historical information.

S2. Substantial and well-measured inference efficiency gains with long histories.

At 256 historical chunks, ReWAM reduces model-side inference latency from 27.47 s to 8.99 s. The measurements include denoising, retrieval, KV gathering, and intermediate cache operations, providing direct experimental support for the computational efficiency benefits of selective historical access.

Critical weaknesses

W1. Insufficient specification of the learning mechanism and validation of register-guided retrieval.

The paper describes how registers are constructed and used, and states that the original video and action prediction objectives are retained. However, the actual optimization process for the newly introduced registers is not sufficiently specified and requires further clarification.

Furthermore, the experiments primarily evaluate final task success without directly verifying whether the retrieval mechanism selects historical evidence relevant to subsequent instructions. The comparison between ReWAM and Local-A also changes both the historical representations and detailed-history access patterns, making it difficult to isolate the independent contribution of instruction-conditioned retrieval.

These limitations affect the reproducibility of the core method and leave the actual contribution of register-guided retrieval and the reasons for its performance variations insufficiently understood.

W2. Insufficient experimental coverage to substantiate the MOSAIC benchmark contribution.

MOSAIC is presented as a benchmark comprising seven task families and eight configurations, yet the paper reports manipulation results only for the three-target and six-target configurations of Store and Retrieve Blocks. The remaining six families are described and illustrated, but no evaluation results are reported for them.

Without evaluation results for the remaining six families, it is unclear how challenging these tasks are or whether they can effectively assess the intended memory capabilities and distinguish between different models. This lack of empirical evidence weakens MOSAIC's contribution as a benchmark.

Questions for authors

Q1. Could the authors clarify how the newly introduced register-based memory and retrieval mechanism is trained in practice, beyond stating that the original video and action prediction objectives are retained? Please explain how the register representations receive training signals and what role the discrete Top-K selection plays during optimization, with sufficient detail to make the method reproducible.

Q2. Could the authors provide direct evidence of retrieval quality, such as selected historical chunks under different subsequent instructions, recall of task-relevant historical evidence, or comparisons with alternative retrieval strategies under a matched historical reading budget?

Q3. MOSAIC defines seven task families, but only one is evaluated. Please report results for the remaining six families, including baseline comparisons, and analyze their difficulty and ability to evaluate different memory capabilities. If these evaluations are not available, please clarify which task families have been implemented and why they have not been evaluated.

Note: The six-target MOSAIC success rates are reported as 45% (ReWAM) and 55% (Local-A) in Section 5.3, but as 46% and 56% in Table 1. Please verify and ensure consistency.

Reviewer_jsYPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces "late-bound manipulation," a setting where a robot accumulates interaction history before receiving the instruction that determines which historical evidence is relevant. The authors propose Mosaic, a benchmark with shared histories paired against divergent subsequent instructions, and ReWAM, a selective-memory world-action model built on LingBot-VA that uses register-guided top-K retrieval to limit detailed historical access during video-action prediction. Experiments compare ReWAM against Dense (full history) and Local-A (dense visual, local action) baselines on RMBench and Mosaic, plus latency scaling measurements up to 256 video chunks.

Critical strengths

The late-bound formulation is genuinely well-motivated. The observation that real deployments often accumulate experience before the task specification arrives—and that this differs from standard memory benchmarks where the goal is known during history formation—is a clean conceptual contribution. The shared-history/divergent-instruction pairing is the right controlled design for isolating instruction-dependent recall, and Figure 1 communicates it effectively.

The latency characterization is honest and directly measured. Reporting full-cycle wall-clock time (including retrieval, KV gathering, and cache operations) rather than estimating from isolated denoising steps is the correct methodology, and the 3.06×/2.81× speedups at 256 chunks are substantiated by per-step breakdowns in Appendix B. The acknowledgment that ReWAM is slower at one chunk and first wins at 16 chunks is a level of candor not always seen.

The register-based memory writing avoids a separate memory encoder and preserves layer-specific historical features while keeping hard top-K selection confined to prediction-time reads. The architectural decision to reuse video-stage selections for the action stage is parsimonious.

Critical weaknesses

The benchmark contribution is undersold relative to its evaluation. Mosaic contains seven task families and eight configurations, yet the paper only evaluates Store and Retrieve Blocks (3- and 6-target variants). The abstract and introduction claim Mosaic "constructs long interaction histories containing multiple candidate events," but the empirical evidence covers one family whose memory demand is a single identity–location binding. Replay Block Trajectory, Audit Tote History, and Calibrate Relay Routes—the tasks requiring ordered segment recall, multi-event conjunction, and relational composition—are exactly the ones that would stress selective retrieval beyond "find the one relevant chunk," and they are absent. The paper explicitly says "we do not report an aggregate over the unevaluated families," which is transparent but leaves the central claim about instruction-dependent recall from long histories supported by a narrow slice. A benchmark paper that evaluates on 1/7 of its own benchmark is a hard sell at ICLR.

The RMBench result undercuts the method's motivation without adequate analysis. ReWAM trails Dense by 4.43 points on average (50.57% vs. 55.00%), and on observe_and_pickup and cover_blocks all policies are near floor. The paper attributes this to a "task-dependent trade-off," but does not test the obvious hypothesis: that these RMBench tasks do not exhibit late-binding (the instruction is available during history), so selection discards information that a dense policy can exploit. If that is the explanation, the RMBench comparison is not evidence against ReWAM—but the paper never establishes it, and instead reports the number and moves on. The result as presented reads as "our method is worse on the standard benchmark," which is a difficult position for a submission whose core claim is that selective access is sufficient.

The Local-A baseline comparison does not isolate the claimed mechanism. Local-A matches ReWAM's action context but retains dense visual history. ReWAM differs from Local-A in two ways: compact registers and selective detailed reads. The paper concedes this ("it does not isolate either component alone") but then draws conclusions about "the ReWAM memory design" as a whole. On Mosaic 6-target, Local-A actually beats ReWAM (55% vs. 45%), which is the one setting where history length and interference are both large—precisely where selection should help most. The paper's explanation is one sentence ("exposes a performance trade-off for selective visual access relative to Local-A") with no ablation of K, retrieval layer, or register contribution. Without a component analysis, the reader cannot tell whether the Mosaic 3-target gain comes from registers, from selection, or from the interaction.

The retrieval design has an unexamined failure mode that the paper's own framing predicts. Section 4.3 selects chunks by query-register scores at layer r, with hard top-K and a fixed K=3. The paper notes hard selection "can miss complementary events" in Limitations, but Mosaic's Audit Tote History and Calibrate Relay Routes are explicitly multi-event conjunction and relational composition tasks—i.e., tasks where the relevant evidence is distributed across chunks and a single top-K score vector may not cover it. Since these tasks are not evaluated, the limitation is asserted but not measured. A reviewer is left without any evidence about how selection degrades as the number of required evidence chunks approaches or exceeds K, which is the most natural question about the method.

Questions for authors

Why were the other six Mosaic task families not evaluated? If they were constructed but not run, what specifically prevented evaluation, and does the shared-history design actually function for tasks like Audit Tote History where the "correct" container depends on a conjunction rather than a single binding?

On RMBench, is the Dense advantage concentrated in tasks where the instruction is available during history formation? A per-task breakdown of whether late-binding holds would determine whether the RMBench comparison is a fair test of the method or a category error.

On Mosaic 6-target, Local-A outperforms ReWAM by 10 points. Does this reverse with a larger K, or does it persist? If it persists at the configuration where selection should matter most, what does that imply about the register-scoring signal?

The retrieval layer r=4 and K=3 are fixed across all settings (Table 2). Were these tuned at all, and on which benchmark? If tuned on Mosaic 3-target, the 93% figure may not reflect a default configuration.

Reviewer_vURbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies robotic manipulation where task instructions arrive after the robot has accumulated interaction history. It introduces MOSAIC, a benchmark pairing shared histories and identical execution-start states with different subsequent instructions, and proposes ReWAM, which combines global registers, instruction-conditioned retrieval of historical detail, and recent context for video and action generation. Experiments show substantially lower long-history inference latency than dense-history baselines, but success-rate improvements vary across tasks, and only one family of the new benchmark is evaluated.

Critical strengths
  1. The problem setting is practically relevant: robots may receive requests after accumulating experiences without knowing the specific task in advance.

  2. The benchmark design is well motivated. Pairing identical histories and execution-start states with different instructions provides a controlled setting for evaluating instruction-dependent use of historical information.

  3. The improvement on the three-target retrieval task is substantial: ReWAM achieves 93% success, compared with 36% for Dense and 67% for Local-A.

Critical weaknesses
  1. The new benchmark is insufficiently validated. MOSAIC introduces seven task families, but only two configurations of one family are evaluated, leaving capabilities such as multi-event composition and event-order recall untested.

  2. Ablations do not isolate the retrieval mechanism. Without register-only or random-retrieval controls, it is unclear whether the gains come from retrieving relevant detail or from information already available through global registers.

  3. The efficiency comparison is incomplete. Success rates and latency for no-history or recent-context-only baselines are missing, making it difficult to assess the benefits and additional computational cost of historical memory.

  4. Performance gains are inconsistent. ReWAM trails Dense on average on RMBench and Local-A on the six-target task, without sufficient analysis of when retrieval helps or fails.

  5. The six-target results are inconsistent: the text reports 45%/55% for ReWAM/Local-A, whereas Table 1 reports 46%/56%.

Questions for authors
  1. Why is evaluation restricted to Store and Retrieve Blocks? Are results available for other task families, particularly those requiring multiple historical events or event-order recall?

  2. How does performance change when selected historical detail is removed, leaving only global registers and local context, or when retrieval is replaced with random selection?

  3. How does ReWAM compare with no-history and recent-context-only baselines in both success rate and inference latency?

  4. Why does ReWAM underperform Local-A on the six-target task? Is the gap primarily caused by retrieval errors or manipulation failures, and does increasing K help?

  5. Do different instructions applied to the same history produce different, relevant retrieved chunks? Can you provide actual retrieval examples or quantitative evidence?

  6. Which six-target results are correct: 45%/55% in the text or 46%/56% in Table 1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.