acceptodds
Under review as a conference paper at ICLR 2027

Real-Time Action Chunking for Relative-Action Robot Policies

Abstract

Relative-action representations offer an appealing action space for robot learning, as they reduce dependence on a specific robot configuration and facilitate scalable robot-free data collection. In such policies, each action chunk is expressed relative to its initial end-effector pose, referred to as the anchor pose. Deploying these policies on robots requires real-time execution for fast and continuous operation. Existing real-time execution methods achieve this by aligning the overlapping actions of consecutive chunks. However, for relative-action policies, such alignment does not ensure consistent absolute target poses because consecutive chunks use different anchor poses. Controller tracking error can further shift these anchors during execution, introducing additional cross-chunk inconsistency. Robot-free demonstrations make this problem more challenging, as they provide no direct supervision for deployment-time tracking errors. In this work, we introduce RRTC, a tracking-error-aware Relative-action Real-Time Chunking framework. RRTC re-expresses the overlapping trajectory of the preceding chunk relative to the robot's realized pose, thereby preserving cross-chunk absolute-target consistency. It further conditions action generation on the tracking error, allowing the policy to adapt accordingly. To enable such training with robot-free demonstration data, RRTC synthesizes pseudo tracking errors from temporal offsets along demonstration trajectories. Experiments across diverse simulated and real-world manipulation tasks show that RRTC enables smoother and more efficient real-time execution while maintaining robust task performance. Source code is available at https://anonymous.4open.science/r/RRTC-ICLR-2141

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Real-Time Robot Execution, Imitation Learning, Action Chunking, Relative-Action Representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bzwwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies real-time execution for chunk-wise relative-pose policies. It argues that aligning overlapping relative actions may fail to preserve the corresponding absolute targets when the robot’s reached pose differs from the preceding chunk’s planned pose. RRTC re-anchors the overlap, conditions suffix generation on tracking error, and trains with pseudo-errors derived from temporal offsets in demonstrations. The authors evaluate it in five simulated tasks under varied execution conditions and on three real-robot tasks. Results and ablations favor RRTC over the reported baselines in the tested settings

Critical strengths
  1. A concrete geometric analysis. The paper distinguishes action-prefix alignment from consistency of the absolute targets represented by those actions, then derives a direct re-anchoring operation.

  2. Useful component ablations and real-robot evaluation. The experiments test multiple execution conditions, and the ablations indicate that re-anchoring, tracking-error conditioning, and pseudo-error training each contribute in the evaluated setting.

Critical weaknesses
  1. The relationship to RTC is not established through a matched comparison. RTC is formulated as generic action-chunk inpainting, while this paper analyzes prior methods using an absolute-target formulation. The reported baselines are TT-RTC and REMAC, including re-anchored variants; the paper does not directly compare against RTC adapted to the same relative-pose representation. The results therefore support RRTC against the reported baselines, but do not establish that RTC itself is limited to absolute-pose actions or cannot be combined with re-anchoring.

  2. The evidence has limited scope. The pseudo-error proxy is evaluated against replay tracking errors, while real-robot evaluation uses one robot, three tasks, and 30 trials per task. It remains unclear how well the proxy covers errors during closed-loop policy execution, particularly when they do not resemble temporal lag.

Questions for authors
  1. How are action coordinates represented in each RTC-related baseline? Can the authors clarify why an inference-time RTC comparison using the same relative-pose representation and re-anchored prefix is not included? Isn't pi0.5 what’s known as a relative pose strategy?

  2. Were tracking-error distributions measured during closed-loop policy rollouts and compared with the pseudo-errors used for training, or is the reported validation based on replayed demonstrations? This would clarify the scope of the pseudo-error results.

  3. Given that this work focuses on the coherent, real-time execution of action chunks, and although the authors have provided the code, are there any more intuitive video demos available to showcase the algorithm's performance?

2 comments

We thank the reviewer for the careful evaluation and insightful comments. We appreciate the recognition of our analysis and validation, and have conducted additional experiments to address the concerns.


Response to W1 and Q1: Inference-Time RTC as a Baseline.

Our original evaluation focused on training-time RTC methods, as TT-RTC has been shown to achieve better performance while avoiding the additional computational overhead of inference-time inpainting guidance, as demonstrated in the TT-RTC paper [1]. Nevertheless, we agree that a direct comparison with inference-time RTC is important for establishing the effectiveness of our RRTC framework.

We add two inference-time RTC baselines using the same relative-action representation: RTC, which directly applies guidance to the preceding chunk's relative-action prefix, and RTC + Re-anchor, which applies guidance to the re-anchored prefix to ensure coordinate consistency. Both baselines are implemented using π0.5\pi_{0.5} with the same chunk-relative end-effector pose representation as RRTC.

Simulation Results. As shown below, RRTC outperforms both RTC and RTC + Re-anchor, achieving the highest average SR (90.3% vs. 85.5% and 87.7%), TPR, and LDLJ. These results demonstrate that re-anchoring inference-time RTC alone is insufficient and support the effectiveness of our RRTC designs.

TaskMetricRTCRTC + Re-anchorRRTC (Ours)
LiftSR ↑99.5%98.0%99.0%
TPR ↑0.4050.3960.401
LDLJ ↑-12.73-12.56-12.56
CanSR ↑94.0%93.5%97.0%
TPR ↑0.1680.1650.170
LDLJ ↑-15.14-15.43-15.12
StackSR ↑99.0%100.0%100.0%
TPR ↑0.1830.1860.198
LDLJ ↑-15.48-15.19-15.28
SquareSR ↑84.0%85.0%88.0%
TPR ↑0.1150.1160.119
LDLJ ↑-16.82-16.70-16.50
MugSR ↑51.0%62.0%67.5%
TPR ↑0.0060.0180.024
LDLJ ↑-19.95-19.85-19.53
AverageSR ↑85.5%87.7%90.3%
TPR ↑0.1750.1760.182
LDLJ ↑-16.02-15.95-15.80

Real-World Results. Since RTC + Re-anchor outperforms RTC in simulation, we focus on RTC + Re-anchor as the stronger baseline for real-world evaluation. Across all three tasks, RRTC achieves higher average SR (88.9% vs. 77.8%), shorter completion time (13.85 s vs. 18.14 s), and better LDLJ (-27.33 vs. -27.84).

TaskMetricRTC + Re-anchorRRTC (Ours)
Pick and PlaceSR ↑100.0%100.0%
Time ↓9.998.23
LDLJ ↑-26.28-26.26
WipingSR ↑73.3%86.7%
Time ↓13.327.75
LDLJ ↑-27.01-25.66
SortingSR ↑60.0%80.0%
Time ↓31.1225.56
LDLJ ↑-30.22-30.07
AverageSR ↑77.8%88.9%
Time ↓18.1413.85
LDLJ ↑-27.84-27.33

Overall, these additional experiments establish a direct comparison with inference-time RTC and show that RRTC provides further improvements beyond re-anchoring alone.

Reference: [1] Black K, Ren A Z, Equi M, et al. Training-time action conditioning for efficient real-time chunking[J]. arXiv preprint arXiv:2512.05964, 2025.


Response to W2 and Q2: Validating Pseudo Tracking Errors under Closed-Loop Execution.

We thank the reviewer for this insightful comment. Our original pseudo-error validation (Appendix B) was based on demonstration replay. To further validate the rationale behind our pseudo-error design, we extend the analysis to closed-loop policy execution across all three real-world tasks, using observations from 140 real-robot execution logs.

Why Temporal Offsets Approximate Tracking Errors. Our pseudo-error design is motivated by the idea that applying different temporal offsets along demonstration trajectories can simulate a range of pose deviations resembling the tracking errors encountered during real-world execution. Specifically, varying the temporal offset κ\kappa generates different relative pose deviations, allowing us to approximate diverse tracking-error conditions without requiring robot-collected error data. Accordingly, we synthesize pseudo tracking errors from robot-free demonstrations as Et,κtr=St−1St+κE_{t,\kappa}^{\mathrm{tr}}=S_t^{-1}S_{t+\kappa}, where κ∈{0,…,4}\kappa\in\{0,\ldots,4\}.

Validation with Closed-Loop Policy Rollouts. To verify whether temporal offsets can approximate tracking errors during actual policy execution, we directly measure the anchor tracking error Et−h,t=Ct−1Xt−h,tE_{t-h,t}=C_t^{-1}X_{t-h,t} at each re-planning step, where CtC_t is the reached pose and Xt−h,tX_{t-h,t} is the preceding chunk's planned target. We analyze observations from 140 real-robot execution logs across all three tasks. Following the same principle as our pseudo-error training, we construct five candidate tracking-error approximations using temporal offsets κ∈{0,…,4}\kappa\in\{0,\ldots,4\} along the recorded preceding-chunk trajectory.

Evaluation Metrics. Following the temporal-offset matching procedure in Appendix B, we report the measured positional tracking error (TE), the residual error after matching the closest temporal offset, and the Temporal-Lag Residual Reduction (TLRR). A high TLRR indicates that temporal offsets approximate the observed tracking error well.

EvaluationMedian TE (mm)Residual (mm)TLRR (%)
Demonstration Replay7.670.4591.80
Towel Pick-and-Place10.850.6393.37
Board Wiping8.860.7690.43
Tabletop Sorting11.031.0188.73
All Closed-Loop Rollouts10.650.8690.26

Results and Implications. As shown above, although closed-loop execution exhibits larger tracking errors than demonstration replay (10.65 vs. 7.67 mm), the same temporal-offset approximation achieves a 90.26% median residual reduction, closely consistent with the 91.8% reported in our original replay validation. Similar results are observed across all three real-world tasks.

These results support the rationale behind our pseudo-error training: sampling different temporal offsets can generate pose deviations that closely approximate the tracking errors encountered during real-world closed-loop execution. Importantly, this conclusion is not limited to demonstration replay but is further explained by real robot policy rollouts.


Response to Q3: Supplementary Video Demonstrations.

We thank the reviewer for this valuable suggestion. We have added a supplementary video showcasing all three real-world tasks: Towel Pick-and-Place, Board Wiping, and Tabletop Sorting. The video includes four baselines (Vanilla π0.5\pi_{0.5}, Temporal Ensemble, TT-RTC, and REMAC), four additional ablation variants, and our full RRTC method. Each method is demonstrated through three repeated real-robot executions per task.

Additional Ablation Variants. The video includes: (1) RTC + Re-anchor, which applies inference-time RTC guidance to a re-anchored prefix; (2) TT-RTC + Re-anchor, which adds coordinate-consistent re-anchoring to TT-RTC; (3) TT-RTC + Re-anchor + Tuned Offset, which further incorporates a task-specific fixed training offset based on measured tracking errors; and (4) TT-RTC + Re-anchor + GlobalConcat, which incorporates tracking-error information through global feature concatenation instead of layer-wise FiLM modulation.

These visual comparisons complement our quantitative results, highlighting the smoother motion, reduced jitter, and faster task execution achieved by RRTC compared with the baselines and ablation variants.

Reviewer_nix9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

RRTC extends real-time action chunking to relative-action policies. It re-expresses the previous chunk using the robot’s actual pose and conditions future actions on tracking error, with training examples constructed from temporal offsets in demonstrations.

Critical strengths

s1. The paper addresses a concrete deployment problem: matching relative actions does not necessarily preserve absolute targets when the robot falls behind its plan.

s2. Real-robot experiments show improvements in execution time, smoothness, and success on the harder tasks.

s3. The ablations include re-anchoring added to existing methods, helping assess the benefit of the additional training components.

Critical weaknesses

w1. The approach is close to TT-RTC with a coordinate correction. The extra training components help in simulation, but their contribution to the real-robot gains is less clear.

w2. The pseudo-error training uses a fixed temporal-offset range. The paper does not show how sensitive the results are to this choice or whether it needs adjustment for different execution settings.

w3. With 30 real-robot trials per task, some success-rate gains are small in absolute counts. On sorting, the improvement over the strongest baseline is only two successes.

Questions for authors

q1. Could you compare against TT-RTC trained with re-anchored prefixes on the real robot, without the explicit tracking-error input?

q2. How does changing the maximum training offset affect performance? Can one setting work across the tested controllers and speeds?

q3. Does RRTC help more when tracking error is larger? Could you compare its gains over TT-RTC at different measured error levels?

q4. Could you add trials on sorting and report confidence intervals to check whether the success-rate improvement is consistent?

4 comments

We thank the reviewer for the thoughtful evaluation and constructive suggestions. We appreciate the recognition of our work's practical value and provide additional experiments to address the remaining concerns.


Response to W1 and Q1: Real-World Contribution Beyond Re-anchoring.

We thank the reviewer for this valuable suggestion. To further evaluate the contribution of our tracking-error-aware design, we conduct additional real-world experiments comparing RRTC against two baselines without explicit tracking-error conditioning: TT-RTC + Re-anchor, which applies coordinate-consistent re-anchoring to standard TT-RTC, and RA-TT-RTC (Tuned Offset), which additionally uses a task-specific fixed training offset corresponding to the mean measured tracking error.

Real-World Ablation Results. We evaluate all methods on three real-world manipulation tasks, with 30 trials per method and task under the same evaluation protocol.

TaskMetricTT-RTC + Re-anchorRA-TT-RTC (Tuned Offset)RRTC (Ours)
Pick-and-PlaceSR ↑100.0%100.0%100.0%
Time ↓10.709.558.23
LDLJ ↑-27.11-27.08-26.26
WipingSR ↑70.0%73.3%86.7%
Time ↓8.708.437.75
LDLJ ↑-25.92-26.07-25.66
SortingSR ↑73.3%76.7%80.0%
Time ↓28.2127.8425.56
LDLJ ↑-30.79-30.63-30.07
AverageSR ↑81.1%83.3%88.9%
Time ↓15.8715.2713.85
LDLJ ↑-27.94-27.93-27.33

As shown above, RRTC outperforms both re-anchored TT-RTC baselines in average SR, completion time, and motion smoothness. Compared with the tuned fixed-offset baseline, RRTC improves average SR by 5.6 percentage points (88.9% vs. 83.3%) while reducing completion time from 15.27 s to 13.85 s. The improvements are particularly evident on Wiping and Sorting, demonstrating that coordinate-consistent re-anchoring and fixed-offset compensation alone cannot fully explain RRTC's performance gains.


Response to W2 and Q2: Sensitivity to Maximum Training Offset.

We thank the reviewer for this suggestion. We conduct additional experiments to evaluate the sensitivity to the maximum training offset κmax⁡\kappa_{\max} and its generalization across different execution settings.

Offset Sensitivity. We evaluate κmax⁡∈\kappa_{\max}\in{0,2,4,8} across five simulation tasks, using two policy seeds and 100 rollouts per seed under the same training and evaluation protocol.

Taskκmax⁡=0\kappa_{\max}=0224488
Lift96.0%98.0%99.0%99.0%
Can98.0%95.5%97.0%96.0%
Stack99.0%99.0%100.0%98.5%
Square82.0%82.5%88.0%80.0%
Mug63.0%67.5%67.5%67.5%
Average87.6%88.5%90.3%88.2%

As shown above, κmax⁡=4\kappa_{\max}=4 achieves the highest average SR (90.3%) and performs well across all tasks. A smaller offset range may fail to cover the tracking errors encountered during execution, while an excessively large range may introduce unrealistic errors that complicate training.

We acknowledge that the appropriate κmax⁡\kappa_{\max} depends on the robot hardware and task motion dynamics. Therefore, its selection should be guided by an analysis of the expected tracking-error distribution. In our setup, we select κmax⁡=4\kappa_{\max}=4 based on temporal-lag coverage analysis, and our empirical results confirm its effectiveness across the evaluated tasks without task-specific tuning.

Generalization Across Controllers and Speeds. As already demonstrated in our original experiments (Figs. 6 and 7, Appendix D.2), we use the same policy checkpoints trained with κmax⁡=4\kappa_{\max}=4 across different controller gains and execution speeds, without retraining or retuning. Specifically, we vary only the controller gain in the controller sweep, while keeping the controller fixed for each task in the speed sweep. The results show that RRTC maintains strong performance across these conditions.


Response to Q3: Performance Gains Under Different Tracking-Error Levels.

We thank the reviewer for this insightful question. To investigate whether RRTC provides greater benefits under larger tracking errors, we conduct an additional analysis across five tasks and four controller gains, covering 4,000 matched episodes per method.

For each task, we independently divide rollouts into low-, medium-, and high-error groups based on the 33.3rd and 66.7th percentiles of their episode-level mean positional tracking errors. The matched TT-RTC + Re-anchor baseline rollouts and our RRTC rollouts are assigned to the same groups, and the results are then aggregated across all five tasks, shown in the following table.

TE LevelMean TETT-RTC + RA SRRRTC (Ours) SRΔSRΔTPR
Low1.53 mm94.6%92.3%-2.3 pp+0.0068
Medium2.05 mm89.5%91.4%+1.9 pp+0.0120
High4.21 mm73.6%86.1%+12.5 pp+0.0291

As shown above, RRTC's advantage increases substantially with tracking-error magnitude. In the high-error group, RRTC improves SR by 12.5 percentage points over TT-RTC + Re-anchor. The TPR improvement also increases from +0.0068 to +0.0291 across the error levels.

When tracking errors are small, anchor shifts introduce only minor cross-chunk inconsistencies, allowing existing methods to achieve comparable performance. However, larger tracking errors are inevitable in real-world execution due to controller limitations and robot dynamics. Under such conditions, RRTC demonstrates substantial improvements, highlighting the importance of our design for robust real-time execution of relative action policies.


Response to W3 and Q4: Additional Real-World Trials and Statistical Reliability.

We thank the reviewer for highlighting this concern. Our real-world evaluation protocol follows established practices in related real-time robot execution studies. Specifically, RTC [1] evaluates 10 trials per task and method, REMAC [2] conducts 30 trials per task and method, and TT-RTC [3] evaluates two real-world tasks with confidence intervals. Following these practices, we conduct 30 real-robot trials per task and method across three manipulation tasks.

To further strengthen our evaluation, we train an additional independent policy seed for the most challenging Tabletop Sorting task, totaling 60 trials per method across two training seeds. The results also include the additional real-world ablations introduced in our response to Q1, as well as a stronger GlobalConcat ablation study, which incorporates TT-RTC with the same tracking-error information through global feature concatenation. We report the SRs for both seeds and the 95% Wilson confidence intervals (CIs) for the combined results.

MethodSeed 1 SRSeed 2 SRCombined SR95% CI
Naive Async66.7%63.3%65.0%[52.4, 75.8]%
Temporal Ensemble50.0%53.3%51.7%[39.3, 63.8]%
TT-RTC70.0%66.7%68.3%[55.8, 78.7]%
REMAC73.3%66.7%70.0%[57.5, 80.1]%
TT-RTC + Re-anchor73.3%70.0%71.7%[59.2, 81.5]%
RA-TT-RTC (Tuned Offset)76.7%76.7%76.7%[64.6, 85.6]%
GlobalConcat76.7%73.3%75.0%[62.8, 84.2]%
RRTC (Ours)80.0%86.7%83.3%[72.0, 90.7]%

As shown above, RRTC consistently achieves the highest SR across both independent training seeds, reaching 83.3% (50/60), compared with 70.0% (42/60) for REMAC, the strongest baseline in our original submission, and 76.7% (46/60) for the newly added tuned fixed-offset baseline. Although the 95% CI for the improvement over the latter still includes zero, RRTC achieves a consistent SR gain of +6.7 percentage points across the combined trials, together with better execution efficiency and motion smoothness.

We also provide a new supplementary video showcasing all three real-world tasks, including comparisons with baselines and additional ablation variants, which further illustrates the smoother and faster execution achieved by RRTC. Together, these results provide complementary evidence supporting the practical effectiveness of our method.

References:

[1] Black K, Galliker M, Levine S. Real-time execution of action chunking flow policies[J]. Advances in Neural Information Processing Systems, 2026, 38: 33383-33407.

[2] Haoxuan Wang, Gengyu Zhang, Yan Yan, Yuzhang Shang, Ramana Kompella, and Gaowen Liu. Real-time robot execution with masked action chunking. In International Conference on Learning Representations, volume 2026, pp. 120161–120179, 2026b.

[3] Black K, Ren A Z, Equi M, et al. Training-time action conditioning for efficient real-time chunking[J]. arXiv preprint arXiv:2512.05964, 2025.

Reviewer_c97xOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RRTC for asynchronous execution of relative-action robot policies. It re-expresses previously committed targets relative to the current measured pose, conditions action generation on tracking error, and synthesizes training errors through temporal offsets in demonstrations. Experiments on five simulation tasks and three real-world tasks report improvements in success rate, execution efficiency, and motion smoothness.

Critical strengths

1.The paper clearly identifies a practical coordinate-consistency issue when applying asynchronous action chunking to observation-relative actions.

2.The evaluation includes controller-gain and inference-delay sweeps, component ablations, and real-world deployment, providing evidence that the combined system improves execution in the tested settings

Critical weaknesses

1.The central contribution appears to correct an improperly adapted baseline rather than introduce a new algorithm.Re-anchoring follows directly from preserving absolute targets when changing the reference pose: Anew=Cnew−1ColdAoldA_{\mathrm{new}}=C_{\mathrm{new}}^{-1}C_{\mathrm{old}}A_{\mathrm{old}}. Expressing this identity as “tracking error × remaining motion” does not make it a new compensation method. Baselines that reuse relative actions without the required coordinate conversion are incorrectly implemented adaptations of RTC to this action representation. Consequently, improvements over these baselines largely demonstrate the consequences of inconsistent coordinates. The included TT-RTC + re-anchoring ablation is the more appropriate baseline and should anchor the main comparisons, particularly on real robots.

2.The remaining novelty is narrow, with modest supporting evidence. Shifting action supervision to accommodate embodiment-specific response delays is an established practical technique, with implementations predating this work in Diffusion Policy and UMI-related code. The remaining contribution is primarily conditioning the policy on errors associated with different temporal offsets. Yet the paper acknowledges that this information is already present in the re-anchored prefix, and its injection reuses existing conditioning machinery. Explicit injection improves average simulation success from 87.5% to 90.3%, but the paper does not establish the consistency of this modest gain across training seeds, demonstrate it through real-world component ablations, or compare against simpler ways of supplying the same information.

3.The speed-scaling protocol is not a realistic model of faster execution on a fixed real-world system.Keeping inference delay at four control steps while increasing control frequency from 20 to 60 Hz implicitly reduces physical inference latency from 200 to approximately 66.7 ms. Increasing robot execution speed does not make an unchanged policy and computing platform perform inference three times faster. Although methods are compared under matched settings at each speed, the sweep simultaneously increases motion demands and relaxes the latency constraint, undermining its interpretation as evidence of robustness to faster real-world execution. Furthermore, the real-world tasks do not establish performance under highly dynamic motion, while pseudo-error validation is limited to four towel demonstrations replayed under four controller gains. Behavior under overshoot, rapid reversals, and non-lag-dominated disturbances remains largely untested

Questions for authors

ow much improvement remains over correctly re-anchored TT-RTC with a tuned, fixed training offset, especially on real robots?

  • How does explicit error injection compare with state concatenation or an additional error token? Can real-world ablations and results across more training seeds establish its independent contribution?- Can the speed sweep be repeated with physical inference latency held fixed? Can the authors evaluate more dynamic tasks and report direct tracking-error and chunk-boundary discontinuity measurements, including conditions where errors cannot be explained primarily by temporal lagow much improvement remains over correctly re-anchored TT-RTC with a tuned, fixed training offset, especially on real robots?

  • How does explicit error injection compare with state concatenation or an additional error token? Can real-world ablations and results across more training seeds establish its independent contribution?- Can the speed sweep be repeated with physical inference latency held fixed? Can the authors evaluate more dynamic tasks and report direct tracking-error and chunk-boundary discontinuity measurements, including conditions where errors cannot be explained primarily by temporal lag?

6 comments

We sincerely thank the reviewer for the detailed and constructive feedback. We appreciate the opportunity to clarify our contributions and provide additional experimental evidence addressing the concerns.


Response to W1 and Q1: Contribution Beyond Coordinate-Consistent Re-anchoring.

We thank the reviewer for the constructive comments. To demonstrate the benefits of our tracking-error-aware design, we compare RRTC against a re-anchored TT-RTC baseline with a tuned fixed training offset and analyze its performance across different tracking-error levels in both simulation and real-world experiments.

Comparison with Tuned Fixed-Offset TT-RTC in Simulation. We construct a stronger baseline, RA-TT-RTC, which combines coordinate-consistent re-anchoring with a tuned, task-specific fixed training offset.

For each task, we select the offset k∗∈k^*\in{0,1,2,3,4} by matching the temporal-offset pseudo-error distribution to measured tracking errors from 100 demonstration replays. We then fine-tune TT-RTC using the same initialization and training budget as RRTC, with two policy seeds and 100 evaluation rollouts per seed.

TaskRA-TT-RTC SRRRTC (Ours) SRΔ\DeltaSR
Lift97.0%99.0%+2.0 pp
Can96.0%97.0%+1.0 pp
Stack99.0%100.0%+1.0 pp
Square86.0%88.0%+2.0 pp
Mug64.0%67.5%+3.5 pp
Average88.4%90.3%+1.9 pp

As shown above, RRTC consistently outperforms the tuned RA-TT-RTC baseline across all five tasks, improving average SR from 88.4% to 90.3%, with the largest gain on Mug (+3.5 pp). These results demonstrate that our tracking-error-aware design provides additional benefits beyond coordinate-consistent re-anchoring and fixed-offset training.

Real-World Component Ablation. We further evaluate RRTC against two stronger baselines: TT-RTC + Re-anchor, which applies coordinate-consistent re-anchoring to standard TT-RTC, and Re-Anchored TT-RTC (Tuned Offset), which additionally uses a task-specific fixed training offset corresponding to the mean measured tracking error. We evaluate all methods across three real-world tasks, with 30 trials per method and task under the same evaluation protocol.

TaskTT-RTC + RA SR ↑Time ↓LDLJ ↑Tuned Offset SR ↑Time ↓LDLJ ↑RRTC (Ours) SR ↑Time ↓LDLJ ↑
Pick-and-Place100.0%10.70-27.11100.0%9.55-27.08100.0%8.23-26.26
Wiping70.0%8.70-25.9273.3%8.43-26.0786.7%7.75-25.66
Sorting73.3%28.21-30.7976.7%27.84-30.6380.0%25.56-30.07
Average81.1%15.87-27.9483.3%15.27-27.9388.9%13.85-27.33

As shown above, RRTC consistently outperforms both re-anchored TT-RTC baselines in average SR, completion time, and motion smoothness. Compared with the tuned fixed-offset baseline, RRTC improves average SR by 5.6 percentage points (88.9% vs. 83.3%), while reducing completion time from 15.27 s to 13.85 s. These results demonstrate that coordinate-consistent re-anchoring and tuned fixed-offset training alone cannot fully explain RRTC's performance gains, supporting the additional benefits of our tracking-error-aware design.

We also provide a supplementary video showcasing all three real-world tasks, including these additional real-robot ablation experiments, to facilitate a more intuitive comparison of execution performance.


Response to W2 and Q2: Independent Contribution of Tracking-Error Conditioning.

Thank you for the insightful comment. Our motivation for tracking-error injection is to provide a direct pathway for the policy to utilize execution-state information that would otherwise need to be inferred from the prefix tokens, enabling more effective adaptation to tracking errors. Our original ablation (Table 1), evaluated with two independently trained policy seeds and 100 rollouts per seed, already demonstrates its independent contribution: removing explicit TE injection reduces the average SR from 90.3% to 87.5%, while retaining re-anchoring and pseudo-error training.

Comparison with Simpler Tracking-Error Conditioning. To further evaluate our injection design, we introduce an additional GlobalConcat baseline, which concatenates the same normalized 8D tracking-error vector with the policy's global conditioning features instead of using layer-wise FiLM modulation. Both methods use identical pseudo-error training, optimization budgets, and controller settings.

TaskGlobalConcat SRRRTC (Ours) SRΔ\DeltaSR
Lift99.0%99.0%0.0 pp
Can93.0%97.0%+4.0 pp
Stack98.0%100.0%+2.0 pp
Square85.5%88.0%+2.5 pp
Mug65.5%67.5%+2.0 pp
Average88.2%90.3%+2.1 pp

RRTC outperforms GlobalConcat by 2.1 percentage points in average SR, suggesting that layer-wise FiLM modulation provides a more effective mechanism for incorporating tracking-error information than simple global concatenation. Together with our original ablation, these results support the effectiveness of explicit tracking-error conditioning beyond re-anchored prefix conditioning alone.

Real-World Results. We further evaluate GlobalConcat on all three real-world tasks, with 30 trials per method and task under the same evaluation protocol.

TaskGlobalConcat SR ↑Time (s) ↓LDLJ ↑RRTC (Ours) SR ↑Time (s) ↓LDLJ ↑
Pick-and-Place100.0%8.48-26.65100.0%8.23-26.26
Wiping80.0%8.42-26.0586.7%7.75-25.66
Sorting76.7%27.75-30.5480.0%25.56-30.07
Average85.6%14.88-27.7588.9%13.85-27.33

As shown above, RRTC achieves higher average SR (88.9% vs. 85.6%), shorter completion time (13.85 s vs. 14.88 s), and better motion smoothness (-27.33 vs. -27.75). We additionally evaluate an independent training seed on Sorting, where RRTC achieves 83.3% SR across 60 trials, compared with 75.0% for GlobalConcat.

Together with our original component ablation, these simulation and real-world results support the effectiveness of the tracking-error injection module, which is more effective than simple global concatenation.


Response to W2 and Q3: Additional Real-World Ablations and Training-Seed Consistency.

We thank the reviewer for this suggestion. To further validate the contribution of our tracking-error-aware design, we conduct additional real-world component ablations across all three tasks and evaluate their consistency across independent training seeds.

Real-World Component Ablations. We compare RRTC with three additional variants: TT-RTC + Re-anchor, which applies coordinate-consistent re-anchoring; RA-TT-RTC (Tuned Offset), which further incorporates a task-specific fixed training offset based on measured tracking errors; and GlobalConcat, which incorporates the same tracking-error information into TT-RTC through global feature concatenation instead of layer-wise FiLM modulation. All methods are evaluated under the same protocol, with 30 trials per task.

MethodPick-Place SR ↑Time ↓LDLJ ↑Wiping SR ↑Time ↓LDLJ ↑Sorting SR ↑Time ↓LDLJ ↑
TT-RTC + Re-anchor100.0%10.70-27.1170.0%8.70-25.9273.3%28.21-30.79
RA-TT-RTC (Tuned Offset)100.0%9.55-27.0873.3%8.43-26.0776.7%27.84-30.63
GlobalConcat100.0%8.48-26.6580.0%8.42-26.0576.7%27.75-30.54
RRTC (Ours)100.0%8.23-26.2686.7%7.75-25.6680.0%25.56-30.07

As shown above, RRTC achieves the best execution time and motion smoothness across all three tasks, while maintaining the highest or tied-highest SR. Its average SR reaches 88.9%, compared with 81.1%, 83.3%, and 85.6% for the three ablation variants, respectively. These results demonstrate that RRTC's real-world performance gains extend beyond coordinate-consistent re-anchoring and fixed-offset compensation. The comparison with GlobalConcat further supports the effectiveness of our layer-wise tracking-error conditioning design.

Additional Training-Seed Evaluation. To further examine training-seed consistency, we train an additional independent policy seed for the most challenging Tabletop Sorting task, covering all baseline and ablation variants, with 30 real-robot trials per method and seed. Due to computational and time constraints, this additional seed evaluation is limited to Sorting.

MethodSeed 1 SR ↑Time ↓LDLJ ↑Seed 2 SR ↑Time ↓LDLJ ↑Combined SR ↑Time ↓LDLJ ↑
Naive Async66.7%39.89-31.4163.3%40.35-31.4865.0%40.11-31.44
Temporal Ensemble50.0%42.72-31.4553.3%41.90-31.3451.7%42.30-31.39
TT-RTC70.0%39.42-31.3766.7%40.05-31.4468.3%39.73-31.40
REMAC73.3%42.39-31.4666.7%41.75-31.3970.0%42.09-31.43
TT-RTC + Re-anchor73.3%28.21-30.7970.0%28.23-30.3971.7%28.22-30.59
RA-TT-RTC (Tuned Offset)76.7%27.84-30.6376.7%27.91-30.3276.7%27.88-30.47
GlobalConcat76.7%27.75-30.5473.3%27.33-30.2875.0%27.54-30.41
RRTC (Ours)80.0%25.56-30.0786.7%25.10-29.9783.3%25.32-30.02

As shown above, RRTC consistently achieves the highest SR, shortest completion time, and best motion smoothness across both training seeds. Over 60 trials per method, RRTC achieves an SR of 83.3%, outperforming the strongest baseline (76.7%), while reducing completion time to 25.32 s and achieving the best LDLJ of -30.02.

Together with our original simulation ablation, where removing explicit tracking-error injection reduces average SR from 90.3% to 87.5%, these additional real-world results provide further evidence that RRTC's tracking-error-aware design offers benefits beyond re-anchoring alone, with consistent improvements across the two independently trained policies evaluated on Sorting.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal and Additional Evidence

We sincerely thank all reviewers for their constructive feedback. To address the major concerns, we have conducted extensive additional experiments and would like to highlight three key takeaways below.

We hope these new results and clarifications have addressed the reviewers' concerns and strengthened their assessment of our work. If so, we would greatly appreciate your consideration of a higher overall rating. We also welcome any remaining questions or feedback.

1. RRTC goes beyond a simple coordinate correction. Re-anchoring alone ensures consistency between overlapping action chunks, but does not guarantee that newly generated actions can effectively handle physical tracking errors. RRTC addresses both through re-anchoring, pseudo-error training, and explicit tracking-error conditioning. Against stronger baselines and ablation variants, RRTC consistently achieves better overall performance. On real robots, its average success rate reaches 88.9%, compared with 83.3% for tuned fixed-offset and re-anchored TT-RTC.

2. The improvements are supported by additional real-world evidence. We conducted component ablations across all three real-world tasks and trained an additional independent seed for Sorting. Across 60 Sorting trials per method, RRTC achieves 83.3% SR versus 76.7% for the strongest new baseline, alongside better execution efficiency and motion smoothness. We also report confidence intervals to clarify the statistical uncertainty.

3. RRTC is beneficial under challenging execution conditions. Its success rate advantage reaches +12.5 percentage points under high tracking errors and +6.8 points at 3× execution speed with fixed physical inference latency. Additional closed-loop measurements, a dynamic manipulation benchmark, and supplementary real-robot videos further support our design.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.