acceptodds
Under review as a conference paper at ICLR 2027

Factored Q-Chunking

Abstract

Reinforcement learning (RL) from demonstrations can teach robots long-horizon manipulation skills despite sparse rewards. Q-chunking, a simple and effective recipe for this setting, has the policy output a short sequence (a chunk) of future actions and scores the whole chunk with one critic. However, it commits every joint of the robot to the same chunk, even though many robots combine parts that act on different time scales, such as a mobile base carrying two arms, or an arm carrying a dexterous hand. Letting some joints re-plan mid-chunk breaks value learning, because a critic over the full chunk then scores actions that are never executed. We propose Factored Q-Chunking (FQC), which commits a slow part of the action for the full chunk and re-plans a fast part halfway through. FQC gives each part its own critic that scores exactly the actions that part executes. On DexJoCo (arm and hand) and BiGym (base and two arms), FQC substantially improves online fine-tuning over Q-chunking under the same policy and budget. Ablations show that matching each critic to what is actually executed is essential.

Primary area:
reinforcement learning
Keywords:
offline-to-online reinforcement learning, action chunking, Q-learning, robotic control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_p9ZBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Factored Q Chunking, which is a variant of Q Chunking that allows the overall action to be factorized into fast and slow partitions being re-planned at different time scales. Each factor of the action is judged by a different critic, but the same actor. The authors show results on two evaluation suites BiGym and DexJoCo as well as ablations with QC and FQC varying partitions and other hyper parameters.

Critical strengths

(S-1) Critic x execution horizon is a nice ablation.

(S-2) Detailed appendix with very transparent statistics and per-task breakdowns. Released code is a plus.

Critical weaknesses

(W-1) I believe that the comparison between FQC at N=32 and QC at N=1 is unfair.

In particular this sentence "FQC improves on Q-chunking on both suites: 0.327 against 0.102 for QCcal on DexJoCo, and 0.480 against 0.302 on BiGym" (line 401). The fair comparison is QCcal+BoN vs. FQC+BoN which makes the gap smaller (0.327 vs. 0.231 (DexJoCo) and 0.480 vs. 0.440 BiGym). In the latter case, the uncertainties overlap meaning FQC is not significantly better.

Can you run FQC without BoN training (FQC with N=1) to compare directly to QC_cal at N=1? It would also be helpful to see N=4 and N=8 as well.

(W-2) The improved average performance of FQC on BiGym seems to be primarily attributed to one task, dsTrays where FQC achieves 0.784 vs. QC_Cal + BoN achieving 0.30 (which brings the average down). I would like to see further mentions of this point and closer ablations of QC_Cal + BoN on this task in particular (since it is the primary reason why FQC is winning on BiGym).

(W-3) In Table 1, it seems like FQC with a random partition is comparable in performance to FQC using slow-fast partitioning. On BiGym it is hard to see what factors matter since all the partition splits are within noise of each other. I would encourage the authors to run more seeds so the results can be statistically significant.

I am willing to raise my score if these concerns are addressed.

Questions for authors

(Q-1) It would be helpful to see a comparison to QC replanning more often at H=4. Can you add a baseline using QC_cal+BoN with H=4 on both suites?

(Q-2) QC_cm controls for parameter count but only at N=1. Can you run QC_cm + BoN?

1 comment
AuthorsOpenReview ↗

Response to Reviewer p9ZB

Thank you for the review and the concrete suggestions. We ran all the experiments that you suggested, and they made the paper stronger. With the same number of candidates, FQC is higher than the single critic at every N on both suites, and the gap on DexJoCo is significant.

W-1. "The fair comparison is QC_cal+BoN vs. FQC+BoN. Can you run FQC with N = 1, and also N = 4 and N = 8?"

We agree that this is the fair comparison, and we ran it at every N. At N = 1 the single critic is QC_cal.

suiteNseedsFQCsingle criticdifference [95% CI]
DexJoCo150.1370.102+0.034[+0.021,+0.048]\color{#1a7f37}{+0.034 [+0.021, +0.048]}
DexJoCo430.3050.227+0.078[+0.006,+0.150]\color{#1a7f37}{+0.078 [+0.006, +0.150]}
DexJoCo830.3240.279+0.045 [−0.082, +0.172]
DexJoCo3250.3270.231+0.096[+0.053,+0.140]\color{#1a7f37}{+0.096 [+0.053, +0.140]}
BiGym130.3520.317+0.035 [−0.174, +0.244]
BiGym430.3860.354+0.032 [−0.058, +0.121]
BiGym830.3850.359+0.027 [−0.176, +0.230]
BiGym3250.4800.440+0.040 [−0.036, +0.116]
  • Without ranking (N = 1), FQC still beats Q-chunking, on all five DexJoCo seeds.
  • At equal N, FQC is higher in all eight comparisons. On DexJoCo the interval excludes zero at N = 1, 4 and 32. On BiGym the gap is within seed noise, as you noted.
  • Line 401, the abstract and the introduction now compare at the same N.

W-2. "The improved average on BiGym seems to be primarily attributed to dsTrays."

We looked closely at this task, and the gain is real. With ten seeds, FQC reaches 0.669 against 0.282 for QC_cal+BoN (+0.387 [+0.054, +0.720], Welch p = 0.025). It is not an effect of ranking either: FQC is higher at every N, and already at N = 1.

dsTrays (3 to 10 seeds)N = 1N = 4N = 8N = 32
single critic0.0950.2170.1570.282
FQC0.7480.7050.4130.669

The four-member critic with ranking (QC_cm+BoN) reaches 0.579. This task does drive the BiGym average: without it the seven-task means are 0.436 for FQC and 0.460 for QC_cal+BoN. Section 5.4 now discusses this.

W-3. "FQC with a random partition is comparable to FQC using slow-fast partitioning. On BiGym it is hard to see what factors matter."

You are right about Table 1: with five seeds, the embodiment split, its swap and the random draw are level on BiGym (0.480, 0.482, 0.477). To test whether that draw is typical, we ran two more random partitions. Both are lower than the embodiment split on every BiGym seed (0.420 and 0.420 against 0.509; for the first, −0.089 [−0.156, −0.021]). On DexJoCo they reach 0.318 and 0.263 against 0.324. So the embodiment split is the best or tied for the best on both suites, and a random partition can be worse.

The factor that matters most on BiGym, and that is statistically significant with five seeds, is that the critics match the execution.

contrast on BiGym (five seeds, paired)difference [95% CI]FQC higher
FQC − partial re-plan (one critic, two horizons)+0.117[+0.048,+0.186]\color{#1a7f37}{+0.117 [+0.048, +0.186]}5/5
FQC − open-loop split (two critics, one horizon)+0.134[+0.035,+0.233]\color{#1a7f37}{+0.134 [+0.035, +0.233]}5/5

The interaction of the two-by-two ablation is +0.211, positive on all five seeds (p = 0.008).

Q-1. "Can you add a baseline using QC_cal+BoN with H = 4 on both suites?"

We added it with five seeds. FQC is higher on both suites.

suiteQC_cal+BoN, H = 4QC_cal+BoN, H = 8FQCFQC − (H = 4) [95% CI]
DexJoCo0.232 ± 0.0390.231 ± 0.0380.327 ± 0.014+0.095[+0.045,+0.146]\color{#1a7f37}{+0.095 [+0.045, +0.146]}
BiGym0.364 ± 0.0400.440 ± 0.0670.480 ± 0.076+0.116[+0.011,+0.221]\color{#1a7f37}{+0.116 [+0.011, +0.221]}

Re-planning every coordinate more often does not help the single critic.

Q-2. "Can you run QC_cm + BoN?"

We ran it with the five seeds of the paper. FQC is higher on both suites.

method (five seeds)DexJoCoBiGym
QC_cm+BoN (N = 32)0.271 ± 0.0390.441 ± 0.092
QC_cal+BoN (N = 32)0.231 ± 0.0380.440 ± 0.067
FQC (N = 32)0.327 ± 0.0140.480 ± 0.076

Doubling the critic recovers less than half of FQC's gain on DexJoCo (+0.040 of +0.096) and none of it on BiGym. Capacity does not explain the result.

New: real-robot validation at equal N. On a Unitree G1 humanoid with an Inspire dexterous hand, FQC succeeds in 40 of 60 evaluation episodes, against 28 of 60 for QC_cal+BoN, both with N = 32. The general comment gives the details.

We hope that the new experiments and the revised text address your concerns. If so, we would be grateful if you would consider raising your score. We are glad to address any concern that remains.

Reviewer_3xPZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Factored Q-Chunking (FQC), which partitions robot actions into slow and fast factors with different execution horizons and trains separate critics with corresponding TD backup horizons. The method also employs best-of-N action selection and value clipping. Experiments on DexJoCo and BiGym demonstrate improvements over standard Q-Chunking, with ablations examining the effects of critic factorization, execution schedules, and action partitioning.

Critical strengths
  1. The paper identifies a meaningful limitation of conventional Q-Chunking and proposes a simple framework for assigning different execution horizons to different action dimensions.
  2. The experiments include useful ablations, particularly the 2×2 comparison of critic structures and execution schedules, which demonstrates the importance of matching critic inputs and backup horizons to actual execution.
  3. The improvements on DexJoCo remain significant against the stronger best-of-N baseline, suggesting that the proposed approach can provide benefits beyond candidate ranking alone.
Critical weaknesses
  1. The action-partition experiments do not convincingly support the motivation. The paper motivates FQC by arguing that different robot components require different control frequencies. However, swapping the slow and fast factors or randomly partitioning the action dimensions achieves comparable performance. This raises questions about whether the proposed factorization actually exploits the different physical timescales of robot components.
  2. Much of the improvement comes from best-of-N action selection. Best-of-N accounts for a substantial portion of the improvement over standard Q-Chunking. Compared with QCcal+BoN, FQC improves success rates from 0.231 to 0.327 on DexJoCo, but only from 0.440 to 0.480 on BiGym, where the difference is statistically insignificant. Furthermore, the improvements are concentrated in a limited number of tasks.
  3. The comparable performance of different action partitions suggests that the improvement may come from using multiple critics with different TD horizons, rather than assigning specific control frequencies to particular robot joints. The paper only considers two action factors. Could the authors investigate whether using more factors and critics with different horizons provides additional benefits? This would help clarify the underlying source of the improvement.
  4. Since FQC relies on ranking 32 candidate action chunks at each decision, its inference cost is potentially higher than that of the original QC-FQL policy. However, the paper does not report inference latency or real-time control performance. This makes it difficult to assess the practicality of the method for real-world deployment.
  5. The paper fixes the slow and fast horizons at 8 and 4 throughout the main experiments. Although some single-horizon baselines are evaluated, the sensitivity of FQC to different horizon configurations remains unclear. Experiments with different chunk lengths would help determine whether the reported gains depend on this particular choice.
  6. The method is specifically motivated by the different control timescales of physical robot components, but all experiments are conducted in simulation. Without real-world experiments, it remains unclear whether the proposed multi-timescale execution strategy provides practical benefits under real-world control latency, observation noise, and physical interactions. Real-world validation would considerably strengthen the paper. I would consider raising my rating if some real-world experimental validation can be provided.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3xPZ (1/2)

Thank you for the detailed and constructive review. Following your suggestion, we validated FQC on a real robot. On a Unitree G1 humanoid with an Inspire dexterous hand, FQC succeeds in 40 of 60 evaluation episodes, against 28 of 60 for the ranked single-critic baseline. We also address each of your other points with new experiments.

ConcernNew experimentResult
W6real-world validationUnitree G1 humanoid with an Inspire dexterous handFQC ahead at every checkpoint: 40/60 vs 28/60
W1does FQC exploit physical time scales?five partitionsthe gain does not depend on them; the embodiment split is the best default
W2is the gain best-of-N?FQC vs the single critic at equal N = 1, 4, 8, 32FQC higher at every N on both suites
W3is it just several TD horizons?two-by-two ablationno: two TD horizons do not help without matched execution
W4inference latencyevery decision timed in the control loopunder 3 ms per decision at 15 Hz
W5other horizonsgrid over (H, H_f)(8, 4) is the best setting on both suites

W6. "Real-world validation would considerably strengthen the paper."

We ran FQC and QC_cal+BoN on a real robot. FQC is ahead at every checkpoint.

checkpoint (20 episodes each)FQCQC_cal+BoN
offline, 200K steps13/208/20
after 50 online episodes10/206/20
after 100 online episodes17/2014/20
total40/6028/60
  • Robot. A Unitree G1 humanoid, standing. The policy controls the right arm (7 joints).
  • Hand. An Inspire dexterous hand (12 joints, 6 motors). The policy controls four motors: thumb flexion, index, middle and ring finger.
  • Task. Pull one tissue out of a pop-up tissue box. The arm moves the hand from its start pose to a point above the box. The thumb and index finger pinch the top tissue. The arm pulls the tissue out to the side, and the hand releases it. An episode succeeds when one whole tissue has left the box.
  • Observation and control. The policy observes the robot state (7 arm joint angles and 4 finger positions) and the position (x, y, z) of the tissue relative to the robot. Control runs at 15 Hz with an 11-D action. The slow factor is the arm and the fast factor is the fingers, as on DexJoCo.
  • Training. Both methods use the same 20 teleoperated demonstrations, the same actor, the same sparse reward, 200K offline steps and N = 32 candidates. We then fine-tune both online on the robot.

Why FQC is better on the robot. An offline probe of the two critics explains the gap. On states where the palm is within 3 cm of the pinch point, the action that FQC selects closes the fingers in 64% of cases, against 44% for QC_cal+BoN. When we change only the finger dimensions of the selected action, FQC's critics always prefer the pinch near the tissue, while the single whole-chunk critic prefers an open hand within 8 cm of it. This is the mechanism of the paper at work: FQC re-selects the fingers every 4 steps with a critic that reads only the fingers.

Videos. Two videos of FQC are in the supplementary material: one episode with the offline checkpoint, and one after online fine-tuning.

The fast horizon fits the hardware. The fingers start to move 73–122 ms after a command, and the fast horizon of 4 steps is 267 ms.

W1. "This raises questions about whether the proposed factorization actually exploits the different physical timescales of robot components."

This is a fair point, and we revised the motivation in the abstract and the introduction. The short answer: FQC does not need the physical time scales to work, and the embodiment split is still the best partition that we found. We added two more random partitions to test this.

pooled successdefaultcomplementaryrandom 1random 2 (new)random 3 (new)QC_cal+BoN
DexJoCo, five seeds0.3270.3140.2920.231
DexJoCo, three seeds0.3240.3180.2630.213
BiGym, five seeds0.4800.4820.4770.440
BiGym, three seeds0.5090.4200.4200.487
  • The gain does not come from knowing which part is slow. On DexJoCo all five partitions, three of them random, reach higher success than the single critic with the same ranking. What they share is that one part of the action re-plans mid-chunk with a critic that matches it.
  • The embodiment split is the best choice. It is the best or tied for the best on both suites, and on BiGym both new random partitions are lower than it on every seed (0.420 against 0.509).

The revised paper therefore presents the time scales of the robot as the reason to split the action and as a natural default for the slow part, and no longer as the source of the gain.

AuthorsOpenReview ↗

Response to Reviewer 3xPZ (2/2)

W2. "Much of the improvement comes from best-of-N action selection."

Ranking helps every method, and we now say so in the abstract. It is also the standard way to select actions in this family: Q-chunking's main agent and DQC both execute the best of N = 32 candidates under their critic, and the DQC paper reports that N = 32 is crucial for its performance (its Figure 5). But ranking does not explain the gain of FQC: with the same number of candidates, FQC is higher than the single critic at every N on both suites.

pooled successN = 1N = 4N = 8N = 32
DexJoCo: FQC0.1370.3050.3240.327
DexJoCo: single critic0.1020.2270.2790.231
BiGym: FQC0.3520.3860.3850.480
BiGym: single critic0.3170.3540.3590.440

On DexJoCo the gap is significant at N = 1, 4 and 32, where FQC is higher on every seed. Two further controls with the same ranking also stay below FQC (0.327 / 0.480): the single critic at H = 4 (0.232 / 0.364) and a capacity-matched four-member critic (0.271 / 0.441).

BiGym. The margin over QC_cal+BoN is smaller on BiGym (+0.040). What BiGym shows clearly is that re-planning needs matched critics. Re-planning with a single critic loses about 0.08 there (partial re-plan: −0.077, on every seed; H = 4: −0.076). FQC re-plans the arms mid-chunk and loses nothing.

Concentration. FQC improves where the tasks leave room. On every DexJoCo task where any method exceeds 15% success (click_mouse, water_plant, hammer_nail), FQC gains 0.15–0.33 over the ranked single critic; the other four tasks are near the floor for every method. On BiGym the largest gain is on dishwasherclosetrays, and it holds with ten seeds (0.669 vs 0.282, p = 0.025).

W3. "The improvement may come from using multiple critics with different TD horizons."

Our two-by-two ablation (Section 5.3) rules this out. The open-loop split has FQC's two critics, with TD horizons 8 and 4 and the same ranking, but executes the whole chunk open-loop. It is below the single critic on both suites (0.205 vs 0.231; 0.346 vs 0.440). Re-planning under one critic does not help either. The gain appears only when the critics match the execution: the interaction is positive on all five seeds of both suites (p = 0.013 and p = 0.008).

The paper also measures why the mismatched designs fail. In the open-loop split, 48 of the 128 executed coordinates of a chunk are read by neither critic. And when we deploy trained QC_cal+BoN checkpoints with the partial re-plan, BiGym success falls from 0.440 to 0.369: a whole-chunk critic ranks re-planned candidates poorly.

More factors. A third factor is a natural extension: it adds a third critic and a second level of re-planning under the same principle. We leave it to future work and now discuss it in Section 6.

W4. "The paper does not report inference latency or real-time control performance."

FQC runs in real time. We timed every decision in the control loop of the robot (RTX 5090, N = 32).

median (ms)99th percentile (ms)maximum (ms)
FQC, decision at the slow boundary0.4881.2101.655
FQC, mid-chunk re-plan0.7081.7452.792
QC_cal+BoN0.4301.1881.412

One control period at 15 Hz is 66.7 ms, so the slowest FQC decision uses about 4% of it, and no decision was late. Ranking is cheap because the actor is a one-step student: the 32 candidates are one batched forward pass, with no iterative sampling.

W5. "The sensitivity of FQC to different horizon configurations remains unclear."

We ran a grid over (H, H_f) with three seeds. (8, 4), which we fixed before running the grid, is the best FQC configuration on both suites, and it matches or exceeds the best single-critic horizon.

BiGymFQC (8, 4)FQC (8, 2)FQC (16, 8)FQC (16, 4)FQC (16, 2)single critic, H = 2 / 4 / 8 / 16
success0.5090.3810.4300.3670.2980.302 / 0.386 / 0.487 / 0.415

On DexJoCo, FQC at (8, 4) is above the single critic at H = 8 (+0.098) and at H = 4 (+0.095 with five seeds), and above FQC at (8, 2).

The horizon matters, as it does for Q-chunking itself. A very long commitment (H = 16) hurts both FQC and the single critic, and with the other pairs FQC is on par with or below the single critic. We therefore recommend (8, 4). The grid is in a new appendix.

Thank you again for the detailed review. We hope that the real-robot results, together with the new experiments above, address your concerns. If so, we would be grateful if you would consider raising your score. If any concern remains, please tell us and we will try our best to address it in the discussion period.

Reviewer_6ZRqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author proposes a new method called FQC where it studies whether re-planning mid-chunk brings performance benefits for action chunking policy. In particular, it uses two critics: a fast critic and a slow critic. The slow critic is the original action chunking critic used in Q-chunking where it estimates the value after a full chunk is committed. The fast critic estimates the value in the middle of the chunk (either with committed chunk or with replanning). Altogether, these two allow the policy to train the 2N-step action chunking Q-function while acting with only N-step action chunks. On 7 DexJoCo and 8 BiGym tasks, the proposed method attains better performance compared to Q-chunking where the Q-function and the policy share the same chunk size.

Critical strengths
  • The proposed method is technically sound and improves upon Q-chunking on 15 simulated tasks (on average).
  • The paper contains careful ablations for each component of the algorithm (but it is unclear all of them are necessary)
Critical weaknesses
  • The experimental results do not compare the proposed method to one of the closest method, DQC in the main paper. The only result in the appendix seems to be run with a lower budget compared to the original DQC paper.
Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6ZRq

Thank you for the review and for pushing us to compare with DQC properly. We now run DQC with its original recipe on all 15 tasks, at our matched budget, and with longer chunks. FQC outperforms DQC in every setting, on every seed. The comparison is in the main text (Table 1 and Section 5.4).

"The experimental results do not compare the proposed method to DQC in the main paper. The appendix result seems to be run with a lower budget."

You were right that the budget matters: DQC is about twice as high with its original recipe as at our budget, in line with the DQC paper's finding that a large batch size is crucial (its Figure 6).

  • Original recipe. The losses of the official code (commit df89825; a unit test checks each loss) with the configuration of the DQC paper for cube-triple and cube-quadruple (its Tables 5 and 7): 1M offline steps at batch size 4096, 1024×4 networks, two critics aggregated by the minimum, N = 32, κ_b = 0.93 and κ_d = 0.8. DQC defines no online phase, so we then add our 500K online steps. We keep our discount (0.99) and use h = 8 and h_a = 4, the critic horizon and the re-planning interval of FQC.
  • Matched budget. DQC at the budget of every other method in the paper.
poolDQC version, readoutDQCFQCDQC − FQC [95% CI]
BiGym (8), 5 seedsmatched budget, 600K0.0490.480−0.431[−0.561,−0.300]\color{#1a7f37}{-0.431 [-0.561, -0.300]}
BiGym (8), 3 seedsoriginal recipe, final0.1250.509−0.384[−0.490,−0.278]\color{#1a7f37}{-0.384 [-0.490, -0.278]}
DexJoCo (7), 3 seedsmatched budget, last five0.0720.324−0.252[−0.282,−0.222]\color{#1a7f37}{-0.252 [-0.282, -0.222]}
DexJoCo (7), 3 seedsoriginal recipe, last five0.1360.324−0.188[−0.269,−0.106]\color{#1a7f37}{-0.188 [-0.269, -0.106]}

FQC is above DQC on every seed in every row. Even the best evaluation along each DQC run (0.243 on BiGym, 0.229 on DexJoCo) is below FQC. Longer chunks do not help DQC: on three BiGym tasks, the original recipe with h/h_a = 16/4 and with 25/5 (the DQC paper's setting for its cube tasks) reaches 0.046 and 0.111, against 0.078 at 8/4 and 0.914 for FQC.

Why does FQC outperform DQC in this setting? The two methods are built for different phases of learning. DQC is an offline method: its paper evaluates it on offline tasks with 3M to 1B transitions, and neither the paper nor the official code has an online phase. Our tasks start from a few demonstrations (2 to 64 per task on BiGym) and are solved by online fine-tuning. The results agree. Offline, DQC with its original recipe is ahead of Q-chunking and FQC. Online, FQC keeps improving and DQC does not. Both methods rank 32 candidates, which is the default of DQC.

pooled success: end of offline phase → end of trainingDexJoCoBiGym
DQC, original recipe (three seeds)0.194 → 0.1360.136 → 0.125
QC_cal+BoN0.086 → 0.2130.014 → 0.440
FQC0.090 → 0.3240.030 → 0.480

This is consistent with how each method obtains its candidates. DQC's critic ranks the samples of a behavior-cloned flow and does not train that flow. In Q-chunking and FQC the candidates come from a student that is trained to maximize the critic, so online reward improves the candidates themselves.

FQC and DQC re-plan differently. One clarification on the summary: FQC does not re-plan the whole action on the short horizon. Only the fast factor (the hand, or the arms) re-plans after 4 steps. The slow factor (the arm, or the base) stays committed for all 8 steps, and each factor has a critic that matches what it executes. DQC re-plans every coordinate on the short horizon. The matching control in our framework, QC_cal+BoN at H = 4, reaches 0.232 on DexJoCo and 0.364 on BiGym with five seeds, against 0.327 and 0.480 for FQC.

"It is unclear all of them are necessary." Two critics and two execution horizons are necessary together. Re-planning the fast factor under one critic does not help, and two critics without re-planning do not help either. Only the matched design gains: the interaction is positive on all five seeds of both suites (p = 0.013 and p = 0.008). The DQC paper reports the same principle: executing only the first part of a chunk without a critic for that part (DQC-naïve) scores 36, against 82 for DQC (its Figure 2).

New: real-robot validation. On a Unitree G1 humanoid with an Inspire dexterous hand, FQC succeeds in 40 of 60 evaluation episodes, against 28 of 60 for the ranked single-critic baseline. The general comment gives the details.

We believe the new comparison fully addresses your concern. We would be grateful if you would consider raising your rating. We are happy to run DQC in any other configuration that you would find informative.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of new results: real-robot validation and new comparisons

We thank all three reviewers for their careful and constructive feedback. During the discussion period we ran a real-robot experiment and seven new sets of simulation experiments. They confirm the main claim of the paper and add real-world evidence. A summary follows; the individual replies give the details.

1. New: FQC on a real humanoid robot. We ran FQC and the ranked single-critic baseline (QC_cal+BoN) on a Unitree G1 humanoid with an Inspire dexterous hand. The task is to pull a tissue out of a tissue box: the arm moves the hand above the box, the fingers pinch the top tissue, and the arm pulls it out. FQC succeeds in 40 of 60 evaluation episodes, against 28 of 60 for QC_cal+BoN, and it is ahead at every checkpoint: offline (13/20 vs 8/20), after 50 online episodes (10/20 vs 6/20) and after 100 online episodes (17/20 vs 14/20). Both methods share the demonstrations, the actor, the training budget and the ranking of 32 candidates. One decision takes under 3 ms at 15 Hz control. Two videos of FQC are in the supplementary material: one episode with the offline checkpoint, and one after online fine-tuning.

2. New simulation results. FQC has the highest success of all compared methods on both suites (pooled success, mean ± s.d. over five seeds unless noted).

methodNDexJoCo (7 tasks)BiGym (8 tasks)
FQC: two matched critics, two horizons320.327 ± 0.0140.480 ± 0.076
open-loop split: two critics, one horizon320.205 ± 0.0150.346 ± 0.055
partial re-plan: one critic, two horizons320.237 ± 0.0310.362 ± 0.076
QC_cal+BoN: one critic, H = 8320.231 ± 0.0380.440 ± 0.067
new QC_cal+BoN at H = 4320.232 ± 0.0390.364 ± 0.040
new QC_cm+BoN: four-member critic320.271 ± 0.0390.441 ± 0.092
new DQC, original recipe (three seeds)320.1360.125
DQC, matched budget (three seeds; five on BiGym)320.0720.049
new FQC without ranking (three seeds on BiGym)10.137 ± 0.0080.352 ± 0.070
QC_cal: Q-chunking with FQC's clamp10.102 ± 0.0070.302 ± 0.027

3. What the new results show.

  • The gain is not an effect of ranking. With the same number of candidates, FQC is higher than the single critic at every N = 1, 4, 8 and 32 on both suites. On DexJoCo the gap is +0.096 at N = 32 and holds on every seed.
  • A shorter horizon or a larger critic does not close the gap. QC_cal+BoN at H = 4 and a capacity-matched four-member critic with ranking both stay below FQC on both suites.
  • FQC outperforms DQC, the closest prior method, on every seed, with DQC's original recipe and at our matched budget.
  • The critics must match the execution. Re-planning part of the action under one critic, or splitting the critic without re-planning, does not help. Only the matched design gains: the interaction is positive on all five seeds of both suites (p = 0.013 and p = 0.008).

4. Changes in the paper. The abstract, the introduction and Section 5.4 now lead with the comparison at equal N, as Reviewers p9ZB and 3xPZ suggested. Table 1 has the new rows, and a new subsection reports the robot experiment. Changed text is blue.

We hope that our responses and the revisions to the manuscript adequately address the reviewers' concerns and provide a clearer understanding of our contributions. We appreciate the reviewers' time and consideration and welcome any further questions or feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.