acceptodds
Under review as a conference paper at ICLR 2027

From Contact to Action: Fold-Contact Affordances for Cloth Unfolding

Abstract

Manipulation depends on physical relationships that visible shape alone can leave ambiguous. Cloth unfolding makes this problem concrete: the same overhead silhouette can conceal different arrangements of folds and support. We study how complementary sensing can expose support cues and make them useful for action. A transparent-table dual-arm setup pairs a top segmentation mask with a bottom contact mask. We derive fold residual, contact overlap, fold-contact frontier, and silhouette-edge maps for an attention encoder and a residual action prior. An online linear, contextual-bandit-style scorer adapts the prior's target ranking from reward, self-supervised unfolding progress, or their mixture. In the long-horizon comparison, raw dual-view pixels improve flatness over a top-only policy; affordance channels reduce folded area by 82.2% relative to raw dual masks in a fast screen; and the reward-bandit bottom-view scorer performs best in the 21-run residual-prior ablation, with within-panel geometry score . These findings suggest a design principle for cloth unfolding: organize perception around support relationships and connect that representation to target selection through interaction feedback.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Learning representations for robotic perception and control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gVYEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how cloth-support information can guide dual-arm unfolding. On a transparent table, morphological operations on a top-view silhouette and a bottom-view near-table mask produce support-residual, contact, frontier, and edge channels for an attention encoder and a residual action prior. SAC learns continuous corrections, while an online linear scorer adjusts grasp-target rankings using reward or unfolding progress. The design is evaluated through fixed-initial-state ablations, randomized initial states, and physical experiments. The contribution is primarily a system-level combination of support-aware representation and action selection, rather than a new reinforcement-learning algorithm or a bandit algorithm with regret guarantees.

Critical strengths
  1. The connection between representation and control is explicit and interpretable. The same parameter-free geometric channels support both state encoding and grasp proposals, rather than merely adding camera input. In the randomized campaign, replacing raw dual-view inputs with dual-view affordances improves the reported F/C from 0.0466/0.766 to 0.0066/0.968. The top-depth control also helps distinguish the role of structured representation from that of additional bottom-view information. This provides useful empirical evidence for task-relevant inductive bias.
  2. The paper meaningfully examines the gap between proxy metrics and actual unfolding. The authors do not equate low F with successful unfolding. The flat-folded-start experiment uses independent resets, coverage measurements, and human fold checks: the full method succeeds in 51/60 trials, versus 36/60 for top-depth and 39/60 for edge-only. Bottom-view noise, physical calibration, and illumination experiments also reveal operating conditions. In particular, the paper reports registration failures and improvements after calibration, rather than showing only ideal perception results.
Critical weaknesses
  1. The core method and controlled experiments remain incompletely specified. This paper does provide the grid, network widths, and scorer hyperparameters. However, reward-component weights, the exact progress-feedback formula, the mixing rule, and adaptive prior gating are not defined. The section titled “Scorer Update and Gating Pseudo-Code,” provides only a high-level update procedure. These are precisely the dimensions varied in Table 3. The paper states that networks and budgets are matched, but does not enumerate the randomized campaign's actual training budget, the prior/scorer modules enabled in each row, or the construction of the top-depth channels. The attribution to representation alone therefore cannot be independently checked from the reported configuration details.
  2. The learned scorer's gains do not yet establish a robust, generalizable advantage. The main ablation uses a shared fixed initial state, only three seeds per variant, and 25k training steps. In Table 16, the heuristic prior obtains a return of −68 for one seed but 31.1 for another, showing that differences in means are sensitive to training variation (Section 4). The additional 75k-step experiment in Appendix L must be taken into account: F decreases from 0.0121 ± 0.0045 to 0.0088 ± 0.0031, consistent in direction but with substantial uncertainty, while return changes only from 24.6 to 25.0 (Table 13). The randomized and physical experiments support the full system, but do not separately test learned versus fixed ranking in those settings (Tables 1 and 5).
Questions for authors
  1. Could you provide complete formulas for the reward, progress feedback, mixed feedback, and adaptive gating, and specify which parameters are updated during physical fine-tuning and whether the scorer continues learning during evaluation?
  2. Which networks, training budgets, and action priors are used for each method in Table 1? How does top-depth estimate contact and frontier channels? Please provide the configurations and uncertainty for each control so that the matched comparison can be verified.
5 comments
AuthorsOpenReview ↗

Gratefully Responding to Your Constructive Feedback: (Part 1/4): Weakness W1, items 1–5 (reward, feedback, and gating formulas)

W1

Thank you for the detailed questions. Notation follows Sec. 3 and Table 24; new symbols are defined where they first appear. The complete formulas are given in our response to Q1, using the same notation.

1. Reward-component weights. Sec. 3 describes the reward only in words. The weights used in the Table 3 runs are: area / spread / flatness gains 300 / 120 / 120; best-area gain 800; coverage / proximity / grasp bonuses 35 / 1.5 / 4; reach 8; flatness bonus 1; action / stall / bad-pull penalties 0.002 / 10 / 120; success bonus 120. The alive term and the direct fold/contact reward terms have zero weight.

2. Exact progress-feedback formula. The grounded geometry progress gtg_t of Sec. 3 is

gt=clip(Gt(10ΔA+5ΔSp−2ΔZ−10ΔF+0.5ΔC),−1,1)g_t = \mathrm{clip}\big(G_t (10 \Delta A + 5 \Delta \mathrm{Sp} - 2 \Delta Z - 10 \Delta F + 0.5 \Delta C), -1, 1\big)

where A,F,C,ZA, F, C, Z are area, folded area, contact ratio and height standard deviation as in Sec. 4, Sp\mathrm{Sp} is the cloth spread (written Sp to avoid a clash with the ranking index SS), and Δx=xt−xt−1\Delta x = x_t - x_{t-1}. The height gate is

Gt=exp⁡(−max⁡(zˉt−ztable−0.02,0)/0.05)G_t = \exp\big(-\max(\bar z_t - z_{\rm table} - 0.02, 0) / 0.05\big)

with zˉt\bar z_t the mean cloth height.

3. Mixing rule. Reward feedback is qt=tanh⁡(Rt/2)q_t = \tanh(R_t/2), as in Sec. 3. Hybrid feedback is

pt=clip(0.75gt+0.25qt,−1,1)p_t = \mathrm{clip}(0.75 g_t + 0.25 q_t, -1, 1)

The geometry term gtg_t is clipped before mixing. The progress-only and reward-only scorers use pt=gtp_t = g_t and pt=qtp_t = q_t.

4. Adaptive prior gating. In the executed action aexec=clip(aRL+λtaprior,−1,1)a_{\rm exec} = \mathrm{clip}(a_{\rm RL} + \lambda_t a_{\rm prior}, -1, 1) of Sec. 3, the prior weight is λt=0.35κt\lambda_t = 0.35 \kappa_t, where κt\kappa_t is the adaptive gate. Let mtfoldm_t^{\rm fold}, mtfrm_t^{\rm fr}, mtconm_t^{\rm con} be the fold-residual, frontier, and contact mask sums divided by the top-mask sum:

nt=clip(max⁡(mtfold/0.04,mtfr/0.02),0,1),ht=clip((mtcon−0.97)/0.03,0,1)n_t = \mathrm{clip}(\max(m_t^{\rm fold}/0.04, m_t^{\rm fr}/0.02), 0, 1), \qquad h_t = \mathrm{clip}((m_t^{\rm con} - 0.97)/0.03, 0, 1)

κt=clip(min⁡(nt(1−0.7ht(1−nt)),1+0.75vlast),0.15,1.25)\kappa_t = \mathrm{clip}\big(\min(n_t (1 - 0.7 h_t (1 - n_t)), 1 + 0.75 v_{\rm last}), 0.15, 1.25\big)

where vlastv_{\rm last} is the most recently stored scorer advantage (AtA_t in Sec. 3).

Because nt[1−0.7ht(1−nt)]≤1n_t[1 - 0.7h_t(1 - n_t)] \le 1, the gate can only attenuate the prior: for any non-empty top mask, κt∈[0.15,1]\kappa_t \in [0.15, 1], i.e. λt∈[0.0525,0.35]\lambda_t \in [0.0525, 0.35]. The advantage term can bind only when vlast<0v_{\rm last} < 0. The upper clipping bound of 1.25 is part of the implementation but is never reached.

Two cases bypass this formula. For an empty top mask (top-mask sum ≤10−6\le 10^{-6}), the fractions are undefined and the gate is set to κt=0\kappa_t = 0, so no prior action is applied. With adaptation disabled, κt=1\kappa_t = 1, i.e. λt=0.35\lambda_t = 0.35 as in Appendix F.

5. Pseudocode and Table 3 conditions. We will replace the high-level Algorithm 1 (Appendix P.1) with explicit feedback, baseline-update, weight-update, gating and reset rules. Table 3 uses learning rates η\eta = 0.10 / 0.20 / 0.40 for the progress / reward / hybrid scorers, each with the adaptive gate disabled or enabled (6 variants), plus the fixed heuristic residual prior: 7 variants × 3 seeds = 21 runs.

AuthorsOpenReview ↗

Sincere Thanks for the Reviewer’s Constructive Comments:(Part 2/4): Weakness W1, items 6–7 and 2×2 experiment; Weakness W2

W1 (continued)

6. The randomized-campaign records (Sec. 4.2) specify 75,000 environment transitions per seed, 3 seeds, aggregated across nine environments. The encoders share feature dimension 128 and channel widths (32, 64, 96).

7. The top-depth affordance variant estimates near-table occupancy from valid top-surface depth with a 5.5 cm threshold, then constructs the same six channels: top, estimated support, fold residual, contact, frontier and edge. Frontier and edge use the same 3×3 morphology as in Sec. 3. This estimate replaces the bottom mask both in the policy observation and in prior target selection. We will document the depth-noise settings and any run-specific overrides.

We ran a 2×2 experiment crossing the policy observation with the action prior, with 5 seeds per condition and paired evaluation on the same 17 held-out states. Reward, SAC configuration, training steps, data split, and checkpoint selection are shared. Both observation types use the encoders of Appendix F, which share the channel widths (32, 64, 96), the four-head bidirectional cross-attention and pooling, and the 128-dimensional output. Only the input stems differ, as the inputs require: top and bottom stems for raw dual masks, and a six-channel global stem plus fold/frontier and contact/edge stems for affordance channels. In C and D the heuristic residual prior is identical and its scorer is frozen at w0w_0.

ConditionPolicy observationAction priorFolded area F ↓ (m²)
ARaw dual masksOff0.0511
BAffordance channelsOff0.0201
CRaw dual masksHeuristic residual prior, fixed scorer (w0w_0)0.0503
DAffordance channelsHeuristic residual prior, fixed scorer (w0w_0)0.0141

This supports the value of the geometric representation within an end-to-end learned policy.

W2

To directly address this concern, we added a controlled comparison between fixed and learned ranking under randomized initial states. All other components and settings are held constant. The fixed scorer uses manually specified, frozen ranking weights, whereas the reward-bandit scorer updates these weights using interaction feedback.

Both variants are trained with randomized initial states for 50k steps using 20 independent training seeds per method without the 100-rollout real-robot fine-tuning. Each trained model is evaluated on the same 100 held-out initial states. Physical evaluation includes three cloth types and 20 trials per method per type, with matched initial configurations, randomized execution order, and balanced use of models from multiple training seeds.

Evaluation metricFixed scorerLearned scorerImprovement [95% CI]
Simulation: folded area F ↓ (m²)0.01480.0067[0.0069, 0.0091]
Physical: coverage gain ↑0.100.13[0.01, 0.05]
Physical: success (NC + fold check)46.7%58.3%[3.5%, 19.8%]
AuthorsOpenReview ↗

We Appreciate the Opportunity to Clarify Based on Your Feedback: (Part 3/4): Question Q1 (complete formulas)

Q1

The following formulas give the complete specification, using the same notation as in W1.

Let Δx=xt−xt−1\Delta x = x_t - x_{t-1}, [x]+=max⁡(x,0)[x\rbrack_+ = \max(x, 0), and clipl,u\mathop{\rm clip}\nolimits_{l,u} denote clipping to [l,u][l, u]. AA, Sp\mathrm{Sp}, ZZ, FF, CC denote projected area, planar spread, height standard deviation, folded area, and contact ratio, respectively.

The reward is

Gt=exp⁡(−[zˉt−ztable−0.02]+0.05)G_t = \exp\left(-\frac{[\bar z_t - z_{\rm table} - 0.02\rbrack_+}{0.05}\right)

Rt=Gt(300ΔA+120ΔSp−120ΔZ+[0.02−Zt]++800ΔAbest)R_t = G_t\left(300 \Delta A + 120 \Delta \mathrm{Sp} - 120 \Delta Z + [0.02 - Z_t\rbrack_+ + 800 \Delta A^{\rm best}\right)

+35Ntcov+8ωt(dt−1−dt)+1.5Ntnear+4Jt\qquad {} + 35 N_t^{\rm cov} + 8 \omega_t (d_{t-1} - d_t) + 1.5 N_t^{\rm near} + 4 J_t

−0.002mean(clip−1,1(at)2)−10Ptstall−120Ptpull+120Itsuccess\qquad {} - 0.002 \mathop{\rm mean}\left(\mathop{\rm clip}\nolimits_{-1,1}(a_t)^2\right) - 10 P_t^{\rm stall} - 120 P_t^{\rm pull} + 120 I_t^{\rm success}

Its auxiliary terms are:

  • ΔAbest\Delta A^{\rm best}: positive improvement over the historical best area, counted only when Gt≥0.5G_t \ge 0.5.
  • NtcovN_t^{\rm cov}: newly crossed A/0.3A/0.3 thresholds {0.70, 0.80, 0.90, 1.00}, requiring Gt≥0.5G_t \ge 0.5, with comparison tolerance 10−610^{-6}. NtnearN_t^{\rm near}: newly crossed gripper-distance thresholds {0.18, 0.12, 0.08} m, summed across arms.
  • For arm r∈{L,R}r \in \lbrace L, R \rbrace, the grasp quality is γr=(1−or)[1−dr/0.15]+\gamma_r = (1 - o_r)[1 - d_r/0.15\rbrack_+, where oro_r is the gripper opening; the arm is active if γr≥0.35\gamma_r \ge 0.35. JtJ_t sums each arm's quality at its first active event, plus one at the first dual-active event. The reach weight is ωt=[1−0.75(γL+γR)/2]+\omega_t = [1 - 0.75(\gamma_L + \gamma_R)/2\rbrack_+, multiplied by 0.2 when both arms are active; dtd_t is the end-effector-to-cloth distance.
  • “New progress” means a grounded improvement over a historical extremum exceeding 2×10−42\times10^{-4} in AA, 10−410^{-4} in Sp\mathrm{Sp}, 2×10−42\times10^{-4} in reduced ZZ, 10−410^{-4} in reduced FF, or 10−310^{-3} in CC, using metres and square metres.
  • Let ktk_t count consecutive active-grasp steps without new progress, resetting otherwise. Then Ptstall=1[kt>6]min⁡(4,(kt−5)/6)P_t^{\rm stall} = \mathbf{1}[k_t > 6]\min(4, (k_t-5)/6), multiplied by 1.5 for a dual grasp. Ptpull=[Δgripper-spant]+P_t^{\rm pull} = [\Delta\mathop{\text{gripper-span}}\nolimits_t\rbrack_+ only during an active grasp without new progress, multiplied by 1.25 for a single grasp.

Progress, reward, and hybrid (mixed) feedback are

gt=clip−1,1(Gt[10ΔA+5ΔSp−2ΔZ−10ΔF+0.5ΔC])g_t = \mathop{\rm clip}\nolimits_{-1,1}\left(G_t [10 \Delta A + 5 \Delta \mathrm{Sp} - 2 \Delta Z - 10 \Delta F + 0.5 \Delta C]\right)

qt=tanh⁡(Rt/2),pt=clip−1,1(0.75gt+0.25qt)(hybrid)q_t = \tanh(R_t/2), \qquad p_t = \mathop{\rm clip}\nolimits_{-1,1}(0.75 g_t + 0.25 q_t) \quad \text{(hybrid)}

Non-finite component signals are set to zero. For the selected feedback ptp_t (gtg_t, qtq_t, or the hybrid), the clipped advantage and EMA baseline are

At=clip−1,1(pt−aˉt),aˉt+1=0.9aˉt+0.1pt,aˉ0=0A_t = \mathop{\rm clip}\nolimits_{-1,1}(p_t - \bar a_t), \qquad \bar a_{t+1} = 0.9 \bar a_t + 0.1 p_t, \qquad \bar a_0 = 0

These are updated only when a valid selection record exists and the scorer learning rate is positive.

For adaptive gating, let mtfoldm_t^{\rm fold}, mtfrm_t^{\rm fr}, mtconm_t^{\rm con} be the fold-residual, frontier, and contact mask sums divided by the top-mask sum:

nt=clip0,1max⁡(mtfold/0.04,mtfr/0.02)n_t = \mathop{\rm clip}\nolimits_{0,1}\max\left(m_t^{\rm fold}/0.04, m_t^{\rm fr}/0.02\right)

ht=clip0,1((mtcon−0.97)/0.03)h_t = \mathop{\rm clip}\nolimits_{0,1}\left((m_t^{\rm con} - 0.97)/0.03\right)

κt=clip0.15,1.25min⁡(nt[1−0.7ht(1−nt)],1+0.75vlast)\kappa_t = \mathop{\rm clip}\nolimits_{0.15,1.25}\min\left(n_t [1 - 0.7 h_t (1 - n_t)], 1 + 0.75 v_{\rm last}\right)

atexec=clip−1,1(atRL+0.35κtatprior)a_t^{\rm exec} = \mathop{\rm clip}\nolimits_{-1,1}\left(a_t^{\rm RL} + 0.35 \kappa_t a_t^{\rm prior}\right)

Here vlastv_{\rm last} is the latest pre-action advantage (the most recently stored AtA_t). The gate κt\kappa_t is zero for an empty top mask or a top-mask sum ≤10−6\le 10^{-6}, and one when adaptation is disabled.

Physical fine-tuning: SAC updates the actor, twin critics, their visual encoders, and the entropy temperature log⁡α\log \alpha (automatic entropy tuning, Appendix F); target critics receive Polyak updates. The learned scorer separately updates its six weights and feedback baseline.

The online scorer continues to update during evaluation, adapting its target ranking weights using interaction feedback within the current trial. At the beginning of each trial, its weights are reset to the same predefined initial values, and its feedback baseline and update counters are cleared. No scorer updates are carried over between trials.

Reviewer_M13sOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies bimanual robotic garment unfolding. The method combines top and bottom views to extract information about how the garment is supported by the table. This information guides grasp selection and robot actions, while an online scorer adapts target selection. Evaluations in simulation and on a real robot suggest that these cues help improve unfolding performance.

Critical strengths
  1. Clear motivation. The top-view silhouette alone may not reveal whether a garment remains folded. Using bottom-view observations to provide additional support information and guide action selection is an intuitive and practically meaningful idea.
  2. Real-world evaluation and discussion of limitations. In addition to simulation comparisons, the paper evaluates real garments and combines coverage with manual fold checks to assess unfolding success. The authors also explicitly discuss limitations related to multilayer stacks, lighting, and camera registration errors.
Critical weaknesses
  1. Some evaluation metrics do not directly establish unfolding success. A garment being close to the table does not necessarily mean that all folds have been removed. Although the real-world evaluation includes independent fold checks, the main scorer comparisons still focus on support and geometric metrics.
  2. Practical deployment remains constrained. The method requires a transparent table and accurate registration between the top and bottom views. Although calibration experiments are provided, the perturbation tests also show that registration shifts can substantially degrade performance.
Questions for authors
  1. Does the online scorer continue to update during evaluation? At the beginning of each trial, are its weights reset to the same initial values, or are the updated weights carried over from the previous trial?
  2. How are the garment instances and initial states used in the 100 real-robot fine-tuning rollouts separated from those used in the final evaluation? Were the garments held out from parameter tuning in Table 5(c) also excluded from this fine-tuning stage?
3 comments
AuthorsOpenReview ↗

Deepest Thanks for Your Constructive Review: (Part 1/2) Weaknesses W1–W2

W1

We agree that near-table support does not by itself establish unfolding. A garment folded flat on the glass has both layers in contact, so F≈0F \approx 0 and CC is high while it is still folded. The paper notes this limitation (Sec. 4.6, L410–412 and L426–427; Appendix V). We therefore use the two kinds of evidence for different purposes:

  • Continuous metrics (AA, FF, CC, z-std) identify which aspect of the cloth state changed. SS is only a within-panel ranking index (L214–215). We do not use them to claim complete unfolding.
  • A trial succeeds only if it reaches the state’s final normalized coverage (NC) standard and passes an independent fold check using the top and bottom images; fff_f is auxiliary (Table 5, Appendix V).

The physical evaluation also targets the failure mode you describe. From a flat double fold, where F≈0F \approx 0 although the garment is still folded, the method reaches 80% success over 20 independent resets (final NC 0.90). On 3 held-out garments starting flat-folded, the full method succeeds in 51/60 trials (85%), versus 36/60 (60%) for top-depth affordance and 39/60 (65%) for edge-only (final NC 0.91 vs. 0.84 for edge-only). These outcomes are scored by NC and the fold check, not by support residual.

For the scorer specifically, we added a physical comparison between fixed and learned ranking:

Both variants are trained with randomized initial states for 50k steps using 20 independent training seeds per method without the 100-rollout real-robot fine-tuning. Each trained model is evaluated on the same 100 held-out initial states. Physical evaluation includes three cloth types and 20 trials per method per type, with matched initial configurations, randomized execution order, and balanced use of models from multiple training seeds.

Evaluation metricFixed scorerLearned scorerImprovement [95% CI]
Simulation: folded area F ↓ (m²)0.01480.0067[0.0069, 0.0091]
Physical: coverage gain ↑0.100.13[0.01, 0.05]
Physical: success (NC + fold check)46.7%58.3%[3.5%, 19.8%]

W2

Thank you for this comment. The method does require a transparent support and calibrated views, but both are small, one-time setup requirements.

  • Transparent table. The support is an off-the-shelf tempered-glass table costing 80 USD, with 30 USD of camera mounts (Appendices C and U). No custom optical bench is required.
  • Calibration. Calibration is a one-time setup step using a double-sided board. The top–bottom calibration depends on the fixed camera and table geometry rather than on the garment or task, so once calibrated, the setup can be used continuously without recalibration as long as the cameras and table are not moved.

After calibration, the median top–bottom offset is below 0.1 policy cell, fewer than 3% of frames exceed one policy cell, mask IoU is at least .97, and the folded-area degradation relative to clean input is at most 1.1× (Table 19).

AuthorsOpenReview ↗

We Appreciate Your Insightful Critique: (Part 2/2) Questions Q1–Q2

Q1

Yes. The online scorer continues to update during evaluation, adapting its target ranking weights using interaction feedback within the current trial. At the beginning of each trial, its weights are reset to the same predefined initial values, and its feedback baseline and update counters are cleared. No scorer updates are carried over between trials.

To assess the effect of this protocol, we conducted an additional comparison using three training seeds and 10 new test initial states per seed. For each seed, all four settings used the same policy checkpoint:

Scorer settingFolded area F ↓ (m²)Contact ratio C ↑
Fixed initial weights; no updates during evaluation0.0144±0.00070.9280±0.0035
Reset each trial; update within trial0.0061±0.00120.9695±0.0060
Carry over across trials; forward test order0.0071±0.00220.9645±0.0110
Carry over across trials; reverse test order0.0070±0.00210.9655±0.0105

Values are mean ± standard error across the three training seeds. All three online-update settings give a lower folded area and a higher contact ratio than the frozen initial weights. The protocol used in the paper (reset each trial, update within the trial) performs best: FF falls from 0.0144 to 0.0061 (−57.6%) and CC rises from 0.9280 to 0.9695. Carrying the weights over across trials brings no further benefit (FF = 0.0071 / 0.0070) and is more variable across seeds, while the forward and reverse test orders give nearly identical results, so the outcome does not depend on trial order.

Q2

Thank you for highlighting this ambiguity. We clarify our evaluation protocol and provide additional experimental results on unseen garment instances below:

1. Strict Separation of Physical Garment Instances

The garment instances used during the 100 real-robot fine-tuning rollouts were strictly disjoint from those used in the final evaluation. Specifically, all three physical garment instances reported in Table 5(c) were completely excluded from real-robot fine-tuning, hyperparameter tuning, and checkpoint selection. This separation was enforced at the level of individual physical garments rather than garment categories.

2. Independent Initial Configurations and Reset Protocol

Initial configurations were also generated independently. Each evaluation trial started from a fresh physical reset without reusing any configurations or trajectories from the fine-tuning rollouts. While fine-tuning and evaluation may share high-level garment categories or initial configuration types, they shared neither physical garment instances nor specific reset states. Furthermore, final evaluation rollouts were never used for policy fine-tuning or model selection. In Table 5(c), each of the three held-out garments underwent 20 independent resets per method (totaling 60 evaluation trials per method), conducted as separately reset trials rather than matched initial states across methods.

3. Additional Evaluation on Unseen Garments (Pre- vs. Post-Fine-Tuning)

To further isolate and evaluate instance-level generalization, we fixed the final checkpoint obtained directly after the original 100 fine-tuning rollouts (without any retraining) and evaluated it on 6 entirely new physical garments across diverse categories that were never seen during fine-tuning or model selection. We conducted 10 independent trials per garment (60 trials per checkpoint) and compared the checkpoints before and after real-robot fine-tuning:

CheckpointSuccess rate
Pre-fine-tuning (baseline)55.0% ± 6.4%
Post-fine-tuning (full method)80.0% ± 5.2%

The fine-tuned policy improves the overall success rate from 55% (33/60) to 80% (48/60), with consistent performance gains distributed across the majority of the unseen garments. This confirms that the improvements learned during real-robot fine-tuning generalize effectively to novel physical garment instances rather than overfitting to the fine-tuning items.

I thank the authors for their response; my concerns have been addressed. To be honest, I am not familiar with this field, so my confidence score remains at 1.

Reviewer_zsgkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses partial observability in cloth unfolding by introducing a dual-view sensing setup: a top-view silhouette mask paired with a bottom-view near-table contact mask obtained via a transparent table. It derives four parameter-free “fold-contact affordance” maps (fold residual, contact overlap, fold-contact frontier, and silhouette edge) that simultaneously serve as policy inputs via an attention encoder and as targets for a residual action prior, whose pick ranking is adapted online with a lightweight linear, contextual-bandit-style scorer. Extensive experiments in simulation and on a real dual-arm system show that the affordance representation and the learned target ranking significantly reduce folded area and improve grounded contact and flatness versus top-only and raw dual-view baselines, with sensitivity analyses and real-world evaluations supporting the practicality and limitations of the approach.

Critical strengths
  1. This paper introduces a simple yet effective dual-view sensing paradigm that exposes support relationships not apparent from top-view alone, instantiated through a transparent-table bottom contact mask. It proposes a clean, parameter-free geometric decomposition into fold-contact affordances that directly operationalize support cues for both perception and action.
  2. This paper provides controlled comparisons: top-only vs raw dual-view vs affordance representation, plus ablations of the residual prior and scorer learning with multiple seeds.
  3. The overall motivation (“support observability for action”) is clearly articulated, and the affordance construction is presented concretely with explicit morphology choices.
  4. It delivers substantial, interpretable gains in fold/contact/flatness that are hard to capture with top-only observables and shows complementary benefits when plugged into existing policies.
Critical weaknesses
  1. Reliance on a transparent-table bottom view limits applicability; the method’s central advantage hinges on a sensing modality not typical in household or industrial settings.
  2. The method explicitly fails on 3–4 layer stacks; the binary near-table mask fundamentally cannot disambiguate layer counts, and the paper stops short of strategies to overcome this (e.g., temporal probing or active perception).
  3. The linear scorer is ad-hoc (no regret guarantees), and while effective empirically, its stability/robustness properties and sensitivity to hyperparameters (e.g., projection bounds) are under-theorized.
  4. The composite “geometry score” S is within-panel normalized and thus not comparable across blocks, making cross-experiment synthesis harder; while absolute metrics are reported, some headline comparisons emphasize S.
  5. Real-world evaluation emphasizes coverage gains and success by initial state; however, detailed fold/contact/flatness measurements on hardware are necessarily indirect and could be expanded with more standardized unfolding success criteria.
  6. Limited discussion of recent generative or model-based approaches for deformable manipulation that target partial observability with multi-view or temporally aggregated signals; a more explicit positioning would strengthen claims about the necessity of bottom-view sensing vs. alternative cues.
  7. Some architectural details (tokenization, attentional exchange specifics, scorer feature construction) are deferred to appendices; a brief in-main-text summary of key encoder hyperparameters would aid reproducibility without appendix hopping.
Questions for authors
  1. How sensitive are results to the choice of εbot and the single-iteration 3×3 morphology? Could a brief ablation (e.g., εbot ±50%, 1–3 iterations, different kernels) show robustness and guide default settings?
  2. Can you report an ablation of the attention-based affordance encoder versus simpler CNN backbones to quantify the benefit of cross-channel attention beyond the affordance construction itself?
  3. How does training time and sample efficiency compare between affordance inputs and raw dual-view (e.g., success/metrics versus steps)? Does the structured representation accelerate learning?
  4. Could you include a comparison to a recent world-model or diffusion-based planning baseline (e.g., normals-based Dreamer or UniClothDiff in a matched setting) to clarify what is gained by complementary sensing versus richer temporal modeling without a bottom view?
  5. What is the runtime breakdown (per-step) for segmentation, affordance map computation, scorer update, and policy inference on the robot? Are there bottlenecks that limit real-time execution?
  6. The real-world evaluation reports coverage gain and success by state. Can you provide additional physical flatness or fold-residual proxies (e.g., height std trends, contact-ratio estimates) to better align with your simulation metrics?
  7. For environments without a transparent table, how viable is the top-depth affordance variant in practice? Do you foresee active interaction (e.g., pokes/lifts) approximating the bottom mask sufficiently to retain most gains?
  8. How robust is the learned scorer to feature scaling or drift over long runs? Did you observe weight saturation or oscillation, and do you recommend re-initialization or annealing strategies?
  9. In multi-layer (3–4 stack) failures, did you observe any consistent visual/top-bottom signatures that could be exploited (e.g., frontier thickness, edge curvature) to detect “unobservable” states and trigger alternate strategies?
5 comments
AuthorsOpenReview ↗

Thank You for Your Valuable Feedback: Response to Reviewer (Part 1/4): Weaknesses W1–W5

W1

Thank you for this comment. The method does require a transparent support and calibrated views, but both are small, one-time setup requirements.

  • Transparent table. The support is an off-the-shelf tempered-glass table costing 80 USD, with 30 USD of camera mounts (Appendices C and U). No custom optical bench is required.
  • Calibration. Calibration is a one-time setup step using a double-sided board. The top–bottom calibration depends on the fixed camera and table geometry rather than on the garment or task, so once calibrated, the setup can be used continuously without recalibration as long as the cameras and table are not moved. All physical trials were run on a single calibration.

W2

We agree. The submitted paper already lists this as a limitation in Sec. 5 and Appendix U.

The failure on 3–4-layer stacks applies to all tested methods and is not specific to ours: it is a limitation of binary occupancy sensing. We do not claim to identify layer count; Sec. 3 states that these channels encode support cues rather than the occluded cloth state. Within this scope, the method handles two-layer and flat-folded states: from a flat double fold it reaches 80% success over 20 independent resets (Table 5a), and on 3 held-out garments starting flat-folded it succeeds in 51/60 trials (85%), versus 36/60 (60%) for top-depth affordance and 39/60 (65%) for edge-only (Table 5c).

The paper proposes two directions (Sec. 5): using the bottom view only as privileged supervision, and using top RGB-D with temporal history or active interaction at inference. These are in line with the reviewer's suggestions. We also ran experiments on them, and none resolves the multilayer case; we will add this result to the revised manuscript, and our future work will focus on this problem.

W3

We agree, and the paper states this explicitly: the scorer is a projected advantage update for target ranking, not a bandit algorithm with a regret guarantee (Secs. 2 and 3). It ranks feasible top-mask candidate cells using the frontier and related features; continuous control is still learned by SAC.

Stability. Appendix P.1 gives a per-step bound: for any feedback, ∥wt+1−wt∥∞≤η+ηρ∥u−l∥∞\Vert w_{t+1} - w_t \Vert_\infty \le \eta + \eta\rho \Vert u - l \Vert_\infty. Projection keeps the weights inside [l,u][l, u] and the decay term anchors them to the heuristic initialization w0w_0, so the scorer cannot diverge.

Hyperparameter sensitivity. Using the policies from three training seeds, we evaluated five shared test initial states per seed. Each condition uses the same policy checkpoint, and the scorer is re-initialized at the start of every episode. Scaling the projection lower and upper bound vectors jointly by 0.5, 1.0, and 1.5 gives the results below; uncertainties are standard errors across the three seed means:

Projected Boundary Scaling FactorSupport Residual Area ↓Contact Ratio ↑
0.50.0073±0.00160.9597±0.0057
1.00.0063±0.00120.9686±0.008
1.50.0059±0.00090.9736±0.0031

W4

We agree. The composite geometry score SS is normalized separately within each comparison panel, so its value depends on the range of that panel, and SS values and their differences are not directly comparable across experiment blocks. Some headline statements emphasized SS, which could be read as a unified performance scale; we will correct this in the revised manuscript.

W5

The current manuscript describes physical success as “meeting the coverage standard and passing an independent human fold check”, but the coverage threshold, the checking procedure, and the failure conditions are not specified in enough detail. The revised manuscript will state:

  • Coverage threshold: NC = final projected area / fully unfolded area of the same garment.
  • Checking procedure: an independent human annotator inspects the top and bottom images for residual folds; a trial counts as a success only if it both meets the NC standard and passes the fold check.
  • Failure conditions: following the current protocol, a trial fails if the success criterion is not met within 720 s, or if at least 50% of the garment area is dragged out of the top camera's view.
AuthorsOpenReview ↗

Appreciation and Response to Reviewer (Part 2/4): Weaknesses W6–W7, Questions Q1–Q2

W6

We agree that recent generative and model-based methods are directly relevant and deserve a fuller discussion. For example, Rome et al. (2026) combine surface-normal inputs with a DreamerV2 world model for in-air cloth unfolding; UniClothDiff (CoRL 2025) uses a canonical garment-mesh prior to reconstruct the state from partial RGB-D observations and combines a diffusion dynamics model with model predictive control for cloth manipulation.

The revised manuscript will make clear that this paper studies the complementary support information provided by the bottom view and how to use it; the bottom view supplies an additional observation constraint and can be combined with these inference-based methods; by itself, it also cannot remove all multilayer fold ambiguity.

W7

We will add a compact main-text specification: six 32×32 input channels; global, fold/frontier, and contact/edge branches; bidirectional four-head cross-attention between the two local branches; and a 128-dimensional visual output. We will also list the six scorer features: frontier, edge, fold residual, top occupancy, contact overlap, and normalized outward distance.

Q1

We ran the requested ablation at the policy level; the protocol and results are given below.

ItemDescription
5 independently trained modelsSame training method and default parameters, trained with 5 different random seeds
20 shared test initial states20 different initial cloth states; all models and all parameter groups use this same set of 20 states
Frozen modelsLearned weights are kept at test time; only εbot\varepsilon_{\rm bot}, the kernel, or the number of iterations is changed, with no retraining
One testOne model performs one unfolding attempt from a specified initial state and the result is recorded; the specified initial state is restored before the next test

All groups use the same frozen models without retraining, and share the same initial states, random seeds, action budget, and scorer initialization; the threshold used to compute the metrics is always 3.5 cm and the metric grid is always 32×32.

Groupεbot\varepsilon_{\rm bot}MorphologyResults
① Default3.5 cm3×3 square kernel, 1 iterationC = 0.9664; F = 0.00659 m²
② Threshold halved1.75 cm3×3 square kernel, 1 iterationC = 0.9571; F = 0.00767 m²
③ Threshold +50%5.25 cm3×3 square kernel, 1 iterationC = 0.9606; F = 0.00717 m²
④ 2 iterations3.5 cm3×3 square kernel, 2 iterationsC = 0.9638; F = 0.00637 m²
⑤ 3 iterations3.5 cm3×3 square kernel, 3 iterationsC = 0.9669; F = 0.00627 m²
⑥ Larger kernel3.5 cm5×5 square kernel, 1 iterationC = 0.9661; F = 0.00661 m²
⑦ Kernel shape3.5 cm3×3 cross kernel, 1 iterationC = 0.9672; F = 0.00665 m²

Q2

To test whether attention adds value beyond the affordance construction itself (with identical affordance inputs), we added a controlled comparison.

It has two groups × three training seeds (six training runs), configured as follows:

GroupInputEncoderOther settings
A: controlSame six affordance channelsPlain CNNSame action prior, same reward-bandit scorer, reward, training budget, and test initial states
B: proposedSame six affordance channelsCurrent attention encoderSame as A

Both groups are retrained with 65,000 transitions per seed; both output 128-dimensional features, use the same validation set for checkpoint selection, and are evaluated in a paired manner on the same test initial states.

MetricA: plain CNNB: attention encoder
Folded area F ↓ (m²)0.007240.00701
Contact ratio C ↑0.9600.962
Height std. Z ↓ (m)0.0200.0178
Return ↑24.3425.69

Under the same affordance input, the attention encoder outperforms the standard CNN across all metrics.

AuthorsOpenReview ↗

Grateful Acknowledgments and Response to Reviewer (Part 3/4): Questions Q3–Q6

Q3

Affordance inputs improve the final geometry metrics.

The latest controlled comparison keeps the SAC configuration, reward, and geometric action prior fixed and changes the policy input, together with the input stems that each input requires (the attention block, channel widths, and output dimension are unchanged; Appendix F). Five training seeds have finished for each condition, each evaluated on six validation initial states.

FF is the folded area (m², lower is better) and CC is the contact ratio (higher is better).

Environment stepsRaw dual-view F ↓Affordance F ↓Raw dual-view C ↑Affordance C ↑
5,9990.09380.05210.530.74
20,9990.064270.03960.6790.802
75,0000.045460.00610.7450.963
  • Training time per step is essentially the same for the two inputs, so the benefit is in sample efficiency, not in cheaper updates.

Q4

Thank you for this suggestion. We agree that a matched comparison with a recent world-model or diffusion based planning baseline would help distinguish the benefits of complementary sensing from those of stronger temporal modeling.

Our motivation video illustrates a relevant ambiguity: a garment can appear unfolded from the top view while folds remain visible from underneath. This motivates directly observing the underside, but does not rule out inferring these folds from observation and action histories using a sufficiently strong model.

We have also started work on this matched comparison. Adapting the baseline to our observation and action interfaces, training it adequately, and evaluating it under a common protocol requires substantial additional work. We currently estimate several weeks to complete a reliable comparison, which extends beyond the rebuttal period. We will incorporate the results into the revised manuscript when available.

Q5

ComponentRuntime
SAM 3 segmentationApproximately 300 ms/frame
Affordance-map morphology0.106 ms median / 0.132 ms p95
Scorer update0.030 ms median / 0.035 ms p95
Policy inference1.338 ms median / 1.671 ms p95

Segmentation is the computational bottleneck. The vision cycle runs at roughly 3 Hz on a local RTX 6000 GPU.

Q6

Additional physical proxies are summarized below.

MetricResult
Final normalized support residual fff_f0.0517
Corresponding mask contact ratio CC0.9483
Normalized residual in the fine-tuned transfer evaluation0.06–0.19
Change in mean-height proxyMedian 10.52 mm

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.