acceptodds
Under review as a conference paper at ICLR 2027

DexHoldem: Playing Texas Hold'em with Dexterous Embodied System

Abstract

Evaluating embodied systems with real dexterous hardware requires more than isolated motor-skill tests: an agent must perceive a changing scene (e.g. a tabletop), choose a context-appropriate action, execute it with a dexterous hand, and leave the scene usable for later decisions. We introduce **DexHoldem**, a real-world benchmark evaluating Texas Hold'em related dexterous manipulations with a ShadowHand. DexHoldem provides 1,470 teleoperated demonstrations across 14 Texas Hold'em manipulation primitives, a standardized physical policy benchmark, and an agentic perception benchmark that tests whether agents can recover the structured game state needed for embodied decision making. On primitive execution, obtains the highest task completion rate (%), while and tie on scene-preserving success rate (%). On agentic perception, Opus 5.5 narrowly leads on both strict problem-level accuracy (%) and average field-wise accuracy (%); the gap between the two exposes the distance between isolated visual sub-capabilities and complete routing-relevant state recovery. Finally, we instantiate the full embodied-agent loop with one agent–policy pairing over 33 closed-loop hand-level rollouts, in which only % of hands complete; retries restore the failed primitive in 12 of 34 dispatches and resolve prolonged execution stalls in three of the four completed hands, which would otherwise have required manual termination. Only one hand completes with neither a retry nor a human-help request. DexHoldem therefore evaluates dexterous tabletop execution, agentic perception, and embodied decision routing in a shared physical setting. Anonymous Website at https://dexholdempage.github.io/DexHoldem.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Benchmark, Embodied Agent, Dexterous Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FZmFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents DexHoldem. It decomposes Texas Hold’em tabletop interaction into 14 manipulation primitives on a ShadowHand–UR platform, providing a series of teleoperated demonstrations, a physical policy benchmark, a state-perception benchmark, and closed-loop system evaluation. The authors also compare nine policies. The main contribution is a benchmark and empirical diagnosis, which is meaningful for the research community.

Critical strengths

S1. This work presents a useful real-hardware benchmark, which is overall comprehensive.

S2. Two-axis scoring exposes side effects hidden by completion alone. Separating task completion from scene preservation, with explicit non-target displacement and overlap criteria, helps assess precision in sequential tabletop manipulation.

Critical weaknesses

W1. Perception scoring omits key runtime information, this limitation weakens the benchmark’s measurement validity.

W2. Ranking stability is insufficiently established. Most primitives receive only five physical trials. The three perception runs reuse the same 36 problems, and the top two configurations differ by just one of 108 attempts. Tables 1–2 provide neither confidence intervals nor variation across policy-training seeds in physical evaluation. The reported differences are descriptive, leaving the stability of benchmark rankings unresolved.

W3. Closed-loop evaluation lacks controls for component attribution. The 33 hands evaluate only GPT 5.5 with π0, without controlled component substitutions or a recovery-disabled condition. The 12/34 successful retries demonstrate that some stalls can be resolved, but do not quantify recovery’s net benefit or separate perception, execution, and routing contributions to hand-level failure. The authors also acknowledge the need for these controls.

W4. A failed pickup automatically becomes a failed paired put-down even when the placement policy is not executed, while Table 11 combines put-down/show. These scores characterize a sequential chain but cannot isolate placement competence. Report both put-down success conditional on successful pickup and independent results from a standardized held-card initial state.

Questions for authors

Q1. Could the authors report held-card rank/suit and scene_stable accuracy, and compare each model with current-image-only input versus annotated-history input? The predecessor-copy baseline does not replace a model-specific history ablation.

Q2. Could the authors compare recovery enabled versus disabled, and an alternative perceiver or ground-truth state input, under matched initial states and common termination rules? Please report completion without human-help requests and failure counts by component. These comparisons would materially affect the assessment of the closed-loop benchmark’s diagnostic value.

Q3. Please report actual execution counts and conditional success rates separately for each put-down and show primitive, and clarify how show trials are scored after a failed pickup. Would an independent evaluation from a standardized held-card state change the comparison of placement competence?

2 comments

Weakness1: Perception scoring omits key runtime information, this limitation weakens the benchmark’s measurement validity.

We agree that the original perception aggregate did not directly score all information required during execution. Held-card recognition and scene-stability checks are already used in our system, and we now report their accuracy separately.

ConfigurationHeld-card exact match (%)Scene-stability balanced accuracy (%)Unstable-scene recall (%)
Opus 5.597.090.3100.0
Sonnet 4.690.991.4100.0
GPT 6 Astra100.076.9100.0
Gemini 3 Flash72.782.793.3

Held-card recognition is evaluated on 11 images over three runs, using only the current image. Scene stability is evaluated on the 36 original scenes with annotated history. We additionally report the without-annotated-history results in our responses to the other reviewers. These evaluations extend coverage to the omitted runtime fields, and we will clarify which capabilities the original aggregate measures and which require separate diagnostics.

The evaluated GPT, Claude, and Gemini configurations use provider-hosted models, so their wall-clock latency includes model inference, provider-side serving, tool execution, and network overhead. This makes it difficult to compare inference efficiency under a shared hardware setup. Our perception scores therefore measure state-recovery accuracy. Deployment latency remains practically relevant, and we acknowledge that the current benchmark does not provide a controlled efficiency comparison.

Weakness2: Ranking stability is insufficiently established. Most primitives receive only five physical trials. The three perception runs reuse the same 36 problems, and the top two configurations differ by just one of 108 attempts. Tables 1–2 provide neither confidence intervals nor variation across policy-training seeds in physical evaluation. The reported differences are descriptive, leaving the stability of benchmark rankings unresolved.

We added 320 physical trials, repeating the full evaluation for π0.5, π0, RDT, and DP (DINO) with the same checkpoints and protocol. Each now has 160 trials: 20 per pickup primitive and 10 per other primitive.

Physical results, with 95% Wilson confidence intervals. SPSR and TCR are reported in percent.

PolicyOriginal batch SPNew batch SPPooled SPSR [95% CI]Pooled TCR [95% CI]
π0.538/8042/8050.0 [42.3, 57.7]59.4 [51.6, 66.7]
π038/8037/8046.9 [39.3, 54.6]55.6 [47.9, 63.1]
RDT24/8026/8031.2 [24.6, 38.8]41.9 [34.5, 49.6]
DP (DINO)21/8021/8026.2 [20.0, 33.6]37.5 [30.4, 45.2]

For these checkpoints, both π policies outperform RDT and DP (DINO) in SPSR (all four pairwise Fisher tests: p ≤ 0.006), but the difference between π0.5 and π0 is not significant (p = 0.65). These repeated evaluations measure physical execution variability; variation across policy-training seeds remains unmeasured.

For perception, three runs still cover only 36 scenes. We therefore report paired comparisons with confidence intervals obtained by resampling scenes, rather than treating all 108 attempts as independent.

Comparison, with annotated historyAccuracy difference in percentage points [95% CI]p
Opus 5.5 (original) vs GPT 6 Astra (re-run)+5.6 [−3.7, +15.7]0.38
Opus 5.5 (original) vs GPT 5.6 Sol (re-run)0.0 [−11.1, +11.1]1.00

Weakness3: Closed-loop evaluation lacks controls for component attribution. The 33 hands evaluate only GPT 5.5 with π0, without controlled component substitutions or a recovery-disabled condition. The 12/34 successful retries demonstrate that some stalls can be resolved, but do not quantify recovery’s net benefit or separate perception, execution, and routing contributions to hand-level failure. The authors also acknowledge the need for these controls.

Question2: Could the authors compare recovery enabled versus disabled, and an alternative perceiver or ground-truth state input, under matched initial states and common termination rules? Please report completion without human-help requests and failure counts by component. These comparisons would materially affect the assessment of the closed-loop benchmark’s diagnostic value.

We agree that the 33 hands evaluated with GPT 5.5 and π0 do not isolate the contributions of perception, execution and routing. Expanding controlled system-level evaluation is currently costly: a complete evaluation session takes over an hour, and perception errors and policy failures frequently require human intervention. These constraints limit the scale of evaluation, but do not resolve the attribution question.

Three of the 33 hands completed without human help, including two that used retries, and 12 of 34 retries completed the stalled primitive. Under a recovery-disabled protocol that terminates at the first stall, truncating the recorded trajectories leaves one unassisted completion out of 33.

Weakness4: A failed pickup automatically becomes a failed paired put-down even when the placement policy is not executed, while Table 11 combines put-down/show. These scores characterize a sequential chain but cannot isolate placement competence. Report both put-down success conditional on successful pickup and independent results from a standardized held-card initial state.

Question3: Please report actual execution counts and conditional success rates separately for each put-down and show primitive, and clarify how show trials are scored after a failed pickup. Would an independent evaluation from a standardized held-card state change the comparison of placement competence?

We do not evaluate put-down independently because it requires the policy to first pick up the card. Providing a pre-held card would bypass this requirement and would not reflect the policy’s performance in the actual task. Moreover, the pickup position is not fixed, so the starting state for put-down depends on the policy’s own grasp. We therefore evaluate put-down after the policy’s own pickup and count a failed pickup as a failure of the paired put-down. The same rule applies to show.

Question1: Could the authors report held-card rank/suit and scene_stable accuracy, and compare each model with current-image-only input versus annotated-history input? The predecessor-copy baseline does not replace a model-specific history ablation.

We now report held-card rank-and-suit exact match, scene-stability balanced accuracy (BA), and model-specific history ablations.

All values below are percentages. “With / without” refers to annotated history. Both conditions retain current and past images. The held-card test uses only the current image.

ConfigurationHeld-card exact matchOverall: with / without annotationsStability BA: with / without annotations
GPT 6 Astra100.043.5 / 38.976.9 / 75.3
GPT 6 Sol90.939.8 / 36.182.3 / 77.2
GPT 5.6 Sol81.849.1 / 25.989.2 / 73.8
GPT 5.6 Luna69.728.7 / 14.885.5 / 71.6
Opus 5.597.049.1 / 36.190.3 / 81.9
Sonnet 5100.036.1 / 21.388.7 / 79.6
Opus 4.7100.034.3 / 20.477.7 / 58.6
Sonnet 4.690.925.0 / 13.991.4 / 50.1
Haiku 4.50.0*13.9 / 4.674.1 / 46.8
Gemini 3 Flash72.720.4 / 13.082.7 / 71.9
Gemini 3.1 Flash Lite100.010.2 / 14.847.8 / 48.9

The held-card test uses 11 images; the state evaluation uses 36 scenes. Each is repeated three times.

Reviewer_hFhEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DexHoldem combines 1,470 accepted demonstrations, 14 manipulation primitives, physical policy evaluation, structured visual-state perception, and closed-loop poker play. The integrated physical setting and attention to scene-preserving success are valuable. The paper also openly shows the difficulty of the task: the evaluated system completes only 4 of 33 hands. As a benchmark contribution, however, its value depends on whether the scoring is reproducible and its comparisons support useful conclusions. The leading policy and perception differences are very small, while the closed-loop evaluation uses only one agent–policy pairing. The current evidence is not yet strong enough for the claimed breadth of conclusions at ICLR.

Critical strengths
  1. An integrated physical benchmark. Combining manipulation, scene preservation, perception, and recovery in one task could expose failures that isolated policy tests miss. That makes the benchmark potentially useful beyond a single model comparison.
  2. Transparent reporting of system limitations. The low hand-completion rate is informative: it shows a substantial gap between component-level scores and end-to-end performance, rather than hiding that gap behind aggregate policy results.
Critical weaknesses
  1. Establish the benchmark’s distinct value. Please compare DexHoldem directly with the closest dexterous-manipulation and embodied-agent benchmarks along the dimensions that matter here: scene disruption, structured-state perception, recovery, and physical evaluation. This would clarify what the poker setting enables scientifically beyond integrating existing components on one robot platform.
  2. Validate the central scoring rules. Scene-preserving success is a useful contribution only if different evaluators would assign it consistently. Please provide annotated borderline cases, scorer agreement, and sensitivity to the displacement and overlap thresholds. These checks are more important than adding further headline rankings.
  3. Do not overinterpret the physical-policy ranking. The two leading policies tie at 38/80 scene-preserving successes; their task-completion counts differ by only three trials (49/80 versus 46/80). Several primitives have only five physical tests each. Please report per-primitive counts and uncertainty, and present these as descriptive results unless a ranking is supported by the data.
  4. Ensure the perception score captures task-critical state. Held-card identity, scene stability, and uncertainty flags are not directly included in the reported aggregate, although they can affect downstream play. A no-vision predecessor-copy baseline also solves 8/36 problems. Please explain the scoring exclusions and separate information obtained from the current image from information recoverable from the supplied history.
  5. Avoid declaring a perception-model winner from a negligible difference. The leading exact-match scores, 49.1% and 48.1%, differ by one of 108 repeated attempts on the same 36 scenes. Their field averages differ by only 0.4 points, and the systems use different harnesses and budgets. Paired scene-level results and a more controlled comparison are needed before this difference can support a model ranking.
  6. Test the claimed value of closed-loop recovery. Four of 33 hands finish, and only one finishes without retry or human help. The traces show that retries can restore progress, but one agent–policy pairing without a no-retry control cannot quantify their benefit. A matched with/without-recovery comparison would make the system-level finding much more persuasive.

I think the most severe problem is that the principal comparisons are too weak to support rankings or recovery claims. The two leading policies tie at 38/80 scene-preserving successes; the leading perception systems differ by one of 108 repeated attempts on the same 36 scenes. The closed-loop study uses one agent–policy pairing without a no-retry control.

Questions for authors
  1. Is scorer agreement or threshold-sensitivity analysis already available for scene-preservation labels? If not, validating the scoring rules would require additional work.
  2. Can you provide paired per-primitive and per-scene outcomes for the leading policy and perception systems, with uncertainty estimates? Would these support the rankings suggested by the point estimates?
8 comments
AuthorsOpenReview ↗

Summary of the Rebuttal

We thank the reviewer for a careful and constructive review. We agree with the central point: the value of DexHoldem as a benchmark rests on reproducible scoring and on conclusions the data actually support, not on headline rankings. We have therefore (i) collected new physical trials, (ii) re-analysed every comparison with uncertainty estimates, (iii) narrowed the recovery claim to what the existing rollouts identify. Summary of new evidence and revisions:

  1. 320 new physical trials. We re-ran the full 80-trial schedule for π0.5, π0, RDT and DP (DINO) with the same checkpoints, protocol and evaluator. Each of these policies now has 160 trials (10 per primitive, 20 per pickup primitive), and the benchmark has 1,040 physical trials in total.
  2. Per-primitive counts and 95% confidence intervals for all leading policies (W3/Q2).
  3. A direct comparison with the closest benchmarks along the four dimensions the reviewer named (W1).
  4. Scoring reliability (W2/Q1): how labels were produced, test–retest stability, a sensitivity bracket for the disruption criterion, a catalogue of borderline cases, and a plain statement of what is still missing.
  5. Recovery (W6): a prefix-matched counterfactual on the 33 existing rollouts that quantifies what retry contributes to unassisted completion.
  6. Perception (W4/W5): about 2,000 new perception trials. They comprise a without-annotated-history ablation over 11 configurations (1,188 trials), re-runs of the four GPT 6 / 5.6 configurations with the original script (432 trials), direct scoring of scene stability, and an image-only held-card diagnostic (363 calls). We report paired scene-level comparisons with scene-resampled CIs.

Weakness1: Establish the benchmark’s distinct value. Please compare DexHoldem directly with the closest dexterous-manipulation and embodied-agent benchmarks along the dimensions that matter here: scene disruption, structured-state perception, recovery, and physical evaluation. This would clarify what the poker setting enables scientifically beyond integrating existing components on one robot platform.

Thank you for this suggestion. We will add the following comparison as a new table. ✓ = evaluated, ◐ = partially evaluated, ✗ = not evaluated.

BenchmarkMulti-finger handPhysical evaluationScene disruption scored separately from successStructured-state perception scoredClosed-loop verification / recovery
ROBEL (Ahn et al., 2020)✓ (3-finger D'Claw)✓✗✗✗
Bi-DexHands (Chen et al., 2022)✓ (Shadow)✗ (sim)✗✗✗
DexArt (Bao et al., 2023)✓ (Allegro)✗ (sim)✗✗✗
DexVerse (2026)†✓ (six hands)✗ (sim)✗ ¹✗✗
TactiDex (2026)†✓ (Inspire)◐ ²✗ ³✗✗
VIMA-Bench (Jiang et al., 2023)✗✗ (sim)◐ ⁴✗✗
CALVIN / LIBERO / RoboTwin 2.0✗✗ (sim)✗✗✗
AutoEval (Zhou et al., 2025)✗✓✗✗✗ ⁵
RoboArena (Atreya et al., 2025)✗✓✗✗✗
EmbodiedBench (Yang et al., 2025)✗✗ (sim)✗◐ ⁶✗
ENACT (Wang et al., 2025a)✗✗ (offline, sim-rendered)✗◐ ⁷✗
MomaGraph-Bench (Ju et al., 2026)✗◐ ⁸✗◐✗
DexHoldem (ours)✓ (ShadowHand)✓ (1,040 trials + 33 hands)✓ (SP/DC/TF/DF)◐ (36 deployment states with and without annotated history, scene stability scored, 11 image-only held-card problems; see W4)◐ (verification, retry and human escalation in the loop; prefix-matched counterfactual, no physical ablation yet; see W6)

† Concurrent work (arXiv, July 2026). ¹ Some success predicates include contact restrictions, but distractor disturbance is not checked. ² Real-world human demonstrations; policies are evaluated in simulation, with qualitative real-robot deployment. ³ Safety metrics concern fingertip contact force, not collateral changes to the scene. ⁴ Constraint-satisfaction tasks (e.g., "sweep without touching") fold a side-effect constraint into a single success bit; disruption is not reported as a separate axis. ⁵ Automatic success detection and scene resets belong to the evaluation infrastructure; they are not a scored capability of the evaluated policy. ⁶ Perception is probed through capability-specific task subsets and scored by task success. ⁷ World-modeling questions over egocentric observations, scored by ordering accuracy. ⁸ Real-robot experiments demonstrate the model; the benchmark itself is evaluated offline.

The table shows that no single existing benchmark covers all four dimensions on physical multi-finger hardware. The reviewer is right, however, that integration alone is not a scientific contribution. The poker setting is useful because of specific properties of the domain:

  • Disruption has a defined downstream cost. In generic clutter, moving a distractor is often harmless, so "side effects" are hard to define. On a poker table, every object encodes game state. A displaced chip changes a stack or a bet; a card left face up leaks private information. Scene preservation is therefore a precondition for the next decision, not an aesthetic criterion, which is what lets us specify disruption rules from task semantics plus geometry. Empirically, 81 of the 364 completions across all 1,040 trials (22.3%, 95% CI [18.3, 26.8]) are disruptive and would count as successes under completion-only scoring. For card reveal by the four leading policies, the figure is 30 of 51 completions (58.8%).
  • The state is discrete and exactly verifiable. Card identities and per-denomination chip counts allow deterministic exact-match scoring against ground truth. This is the same structured state that the router consumes in the closed loop, so the perception benchmark and the system evaluation share one state definition.
  • Dexterous difficulty varies sharply within one scene. Thin (~0.3 mm) cards and stacked chips span very different contact regimes. Pickup is nearly solved by the π policies (39/40 and 35/40 scene-preserving), while single-chip push/pull remains at 23/80 and 21/80 (p < 10⁻⁹ within each policy). This within-scene spread is what makes component failures traceable.
  • Dependencies compound. Pickup → read → put-down, and bets decompose into per-chip pushes. Component scores can therefore be related to hand-level outcomes: roughly 50% primitive SPSR for the top policies versus 12.1% hand completion.

Weakness2: Validate the central scoring rules. Scene-preserving success is a useful contribution only if different evaluators would assign it consistently. Please provide annotated borderline cases, scorer agreement, and sensitivity to the displacement and overlap thresholds. These checks are more important than adding further headline rankings.

Question1: Is scorer agreement or threshold-sensitivity analysis already available for scene-preservation labels? If not, validating the scoring rules would require additional work.

We agree that scene-preserving success is only useful if it is assigned consistently. Below we describe how labels were produced and report the reliability evidence we can provide now.

How labels were produced. All 1,040 physical trials were labeled on site by the same trained evaluator, after the robot and all objects had settled. The evaluator applied the written rubric in Section A.2, which was fixed before evaluation began and was not changed afterwards. The rubric is designed to minimise judgment calls:

  1. completion and disruption are decided independently;
  2. the playable-region rule triggers only when an object falls off the table or its entire footprint leaves the calibrated mask;
  3. overlap is a categorical (yes/no) condition, excluding pre-existing stacks and task-required contacts;
  4. displacement tolerances scale with object size (one radius for chips, half the shortest side for cards) and include in-plane rotation;
  5. temporary occlusion by the hand is ignored;
  6. every primitive has its own explicit completion endpoint (Table 9).

Weakness3: Do not overinterpret the physical-policy ranking. The two leading policies tie at 38/80 scene-preserving successes; their task-completion counts differ by only three trials (49/80 versus 46/80). Several primitives have only five physical tests each. Please report per-primitive counts and uncertainty, and present these as descriptive results unless a ranking is supported by the data.

We agree, and we have revised the text accordingly. To support a more careful reading, we doubled the physical evaluation of the four leading policies (320 new trials; same checkpoints, protocol and evaluator).

Aggregate results with 95% Wilson intervals

PolicyBatch 1 (paper) SPBatch 2 (new) SPPooled SPSR (n=160)Pooled TCR (n=160)
π0.538/8042/8050.0 [42.3, 57.7]59.4 [51.6, 66.7]
π038/8037/8046.9 [39.3, 54.6]55.6 [47.9, 63.1]
RDT24/8026/8031.2 [24.6, 38.8]41.9 [34.5, 49.6]
DP (DINO)21/8021/8026.2 [20.0, 33.6]37.5 [30.4, 45.2]

Per-primitive outcome counts (pooled; SP/DC/TF/DF)

Primitivenπ0.5π0RDTDP (DINO)
pick_up_left2019/0/1/018/1/1/015/1/4/010/2/8/0
pick_up_right2020/0/0/017/0/3/012/0/5/38/0/12/0
push_5103/0/7/03/0/7/03/2/5/06/1/3/0
push_10104/0/6/03/0/7/01/0/9/01/1/8/0
push_50102/1/7/03/1/6/00/0/10/01/2/7/0
push_100103/1/6/02/1/7/04/0/6/04/1/5/0
pull_5103/1/5/12/1/5/21/0/9/00/3/6/1
pull_10102/1/6/12/0/8/00/1/9/03/0/7/0
pull_50103/0/7/04/0/6/00/0/10/00/1/5/4
pull_100103/1/6/02/0/8/02/0/8/02/0/7/1
put_down_left107/1/1/18/2/0/04/0/6/02/1/7/0
put_down_right106/2/2/05/0/5/02/4/4/01/0/9/0
show_left101/5/4/04/5/1/04/5/1/02/5/3/0
show_right104/2/4/02/3/5/02/4/4/02/1/7/0
Total16080/15/62/375/14/69/250/17/90/342/18/94/6

As in the paper, a put-down/show trial whose seeding pickup failed counts as a failure. This applies to 1/40 such trials for π0.5, 4/40 for π0, 12/40 for RDT and 20/40 for DP (DINO). π0.5 and π0 are not separable. A tier-level ordering is supported and reproduces.

Reviewer_DJRNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DexHoldem is a real-world benchmark for dexterous card and chip manipulation using a ShadowHand. It provides 1,470 demonstrations across 14 primitives, evaluates nine policies using task-completion and scene-preservation criteria, and compares perception models on 36 tabletop states. A further 33 closed-loop rollouts examine full-hand execution and recovery.

Critical strengths
  1. The real-world dataset and 720 physical policy trials represent substantial experimental effort and provide a useful resource for dexterous manipulation.

  2. Separating primitive completion from scene preservation captures consequential failures that ordinary success rates would miss.

Critical weaknesses
  1. The task diversity is limited. Several of the 14 primitives differ mainly in direction or chip denomination, and evaluation remains within one tabletop setup. The benchmark’s broader value beyond this specific configuration is not sufficiently demonstrated.

  2. The perception benchmark is small and only partially represents deployment demands. It supplies annotated history and does not directly score held-card identity or scene stability, although both are important to the execution loop.

  3. The component benchmarks are not convincingly connected to system-level performance. Only one agent–policy pairing is evaluated in closed loop, without controlled component substitutions or a recovery ablation. The results therefore offer limited guidance on which improvements would increase autonomous completion.

Questions for authors
  1. How well do the policies generalize when card appearances, chip layouts, or target positions differ from the demonstration setup?

  2. How does perception performance change without annotated history and when held-card identity and scene stability are included?

  3. Does improving the perceiver or policy improve full-hand completion, and how much does recovery contribute under a controlled comparison?

4 comments

We greatly appreciate the reviewer's constructive suggestions. Below are our responses to the questions and weaknesses you raised.

Weakness1: The task diversity is limited. Several of the 14 primitives differ mainly in direction or chip denomination, and evaluation remains within one tabletop setup. The benchmark’s broader value beyond this specific configuration is not sufficiently demonstrated.

We agree that the 14 primitives include directional and denomination variants and are not 14 distinct task families. However, DexHoldem targets precise, scene-preserving dexterous control within a defined Texas Hold’em setting. The 14 primitives cover the manipulation operations required by our evaluated game protocol.

Even with limited coverage of cross-environment generalization, DexHoldem tests several key capabilities: contact-rich hand–card manipulation, accurate perception of chips in the complex scene, and scene-preserving ability. Together, these demands provide a focused benchmark of dexterous perception and control.

Weakness2: The perception benchmark is small and only partially represents deployment demands. It supplies annotated history and does not directly score held-card identity or scene stability, although both are important to the execution loop.

Held-card recognition and scene-stability checks are already part of our experimental system. §A.1 (lines 801–806) specifies waiting while the scene is unstable and recording the held card’s rank and suit before put-down. §B.3 (lines 1551–1555) documents scene-related waiting and card-reading steps. Table 2's eight-field aggregate did not directly score these runtime checks. We acknowledge this coverage gap and now report them separately.

Direct held-card and stability scores (%).

ConfigurationHeld-card exactStability BAUnstable recall
Opus 5.597.090.3100
Sonnet 4.690.991.4100
GPT 6 Astra*10076.9100
Gemini 3 Flash72.782.793.3

Also, we append the results without annotated history/caches (%).

ConfigurationOverallAvgStability BAUnstable recall
Opus 5.536.172.381.980.0
Sonnet 4.613.931.250.16.7
GPT 6 Astra38.978.675.3100
Gemini 3 Flash13.038.971.960.0

These descriptive results extend evaluation to task-critical runtime fields and support a fuller assessment of perception under both context settings.

Weakness3: The component benchmarks are not convincingly connected to system-level performance. Only one agent–policy pairing is evaluated in closed loop, without controlled component substitutions or a recovery ablation. The results therefore offer limited guidance on which improvements would increase autonomous completion.

DexHoldem connects component evaluation to system-level behavior through shared endpoint criteria and recorded intervention reasons. For card placement, the primitive benchmark requires release face down at the requested side (§A.2, lines 1132–1134). In closed-loop play, 11 of 18 human-help requests (61.1%) reported cards returned face up after put-down (§4.5, lines 465–467). Thus, a failure explicitly captured by the primitive criterion is also a frequent reason for intervention during a hand. This identifies a concrete improvement priority: more reliable face-down placement. Its downstream effect can be tested by substituting only the placement policy while fixing perception, other primitives, routing, and recovery, then comparing intervention frequency and unassisted hand completion under matched initial conditions. Separately, 12 of 34 dispatched retries completed the retried primitive (§4.5, lines 458–459), documenting local recovery of progress. These observations provide diagnostic guidance, the current results identify consequential failure modes and specific interventions to test.

Question1: How well do the policies generalize when card appearances, chip layouts, or target positions differ from the demonstration setup?

Appendix A.2 describes how we varied the starting configuration within the tabletop layout. Card trials use designated robot-left and robot-right targets. For each chip denomination, the five push trials contain one through five chips, changing the number of distractors around the target. The five pull trials start from a layout containing chips on both sides and vary which chips remain on each side. In each five-trial chip schedule, three trials use standard chips and two use taped chips to vary surface friction.

Question2: How does perception performance change without annotated history and when held-card identity and scene stability are included?

Removing annotated history lowers overall accuracy for 10 of 11 configurations. We also directly evaluate held-card identity and scene stability. Representative results are shown below; all values are percentages.

ConfigurationOverall: with / without annotationsHeld-card exact matchStability BA: with / without annotations
GPT 6 Astra*43.5 / 38.9100.076.9 / 75.3
Opus 5.549.1 / 36.197.090.3 / 81.9
Sonnet 4.625.0 / 13.990.991.4 / 50.1
Gemini 3 Flash20.4 / 13.072.782.7 / 71.9

BA denotes balanced accuracy. Removing annotations retains current and past images; the held-card test uses only the current image and requires the correct rank and suit. Held-card and stability scores are reported separately from Overall.

Question3: Does improving the perceiver or policy improve full-hand completion, and how much does recovery contribute under a controlled comparison?

For recovery, we can answer a narrower question using the recorded hands. Of the four completed hands, one completed without retry or human help, two completed with retry alone, and one completed with both retry and human help. Across all 33 hands, 12 of 34 dispatched retries completed the stalled primitive.

Dear Reviewers,

Thank you again for your time and thoughtful feedback. We would greatly appreciate your thoughts on our rebuttal, particularly whether our responses have addressed your concerns and whether any points require further clarification.

We believe that Texas Hold’em offers an interesting setting for studying the connection between high-level reasoning and dexterous robot execution, and we would very much welcome your further suggestions on this work. We hope to continue the discussion and are happy to answer any remaining questions or clarify our responses.

Best regards,
The Authors

Dear Reviewers,

I hope this message finds you well. As the discussion period is nearing its end with less than one day remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Comments on the paper

7 comments

Dear Reviewers, Area Chair, Senior Area Chair, and Program Chairs,

Thank you for your time and effort in evaluating our work and for your constructive feedback. We sincerely appreciate your contributions to the review process.

We kindly invite you to read our rebuttal and share any further questions, concerns, or suggestions. We welcome continued discussion and would be happy to provide additional clarification.

If our responses have resolved the reviewers’ concerns, we would be grateful if they could reconsider their initial assessments and, where appropriate, revise their scores upward.

Thank you again for your consideration and support.

Best regards, The authors

Thank you for the detailed response. It overall clarifies the mentioned issues. I keep the previous positive rating.

Thank you very much for your response! If at all possible, I would still appreciate it if you could consider increasing the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.