acceptodds
Withdrawn from ICLR 2027

TETRIS: Self-Supervised Task Representation Learning for Point In-Context Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Point In-Context Learning (Point ICL) aims to infer a transformation from a support pair and apply it to a query point cloud without task-specific parameter updates. However, existing protocols train and evaluate models on the same predefined task families, allowing a task-recognition shortcut: models may identify and execute a familiar task rather than infer the transformation from context. Such evaluations leave transfer to unseen transformation processes unresolved. To assess such transfer, we introduce PIC-Bench, a benchmark of five structured point cloud restoration tasks modeling corruptions arising in 3D acquisition and data processing, spanning both deterministic and ambiguous transformations. PIC-Bench reveals a substantial generalization gap in existing Point ICL methods, suggesting that success on familiar tasks does not readily transfer to unseen transformation processes. To address this limitation, we propose Task rEpresenTation leaRnIng through Self-supervision (TETRIS), a self-supervised task representation learning framework requiring no training pairs from downstream tasks. TETRIS uses unlabeled clean point clouds as targets and generates corresponding sources through online compositions of parameterized atomic point-set operators. Applying the same composition across different objects encourages the model to represent transformations independently of object geometry. On PIC-Bench, TETRIS achieves the best performance across all five tasks and reduces the overall Chamfer distance from to compared with the same backbone trained under the original supervised protocol. When fine-tuned on downstream ShapeNet-In-Context tasks, TETRIS pre-training further improves data efficiency, reducing the Chamfer distance from to with only of the labeled data.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Point Cloud Understanding, In-Context Learning, Representation Learning, Self-supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_hmPCOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits Point In-Context Learning (Point ICL) and argues that existing protocols may conflate true task inference with task recognition, since training and evaluation typically share the same predefined task families. To better evaluate transfer to unseen transformation processes, the authors introduce PIC-Bench, which contains five structured point cloud restoration tasks, and propose TETRIS, a self-supervised framework for learning task representations from unlabeled clean point clouds. TETRIS constructs training episodes by composing parameterized atomic point-set operators online and applying the same transformation to different objects, encouraging the model to capture transformation-related information rather than object-specific geometry. On PIC-Bench, TETRIS pretrained on OpenShape achieves an overall Chamfer distance of 8.7, substantially outperforming DeformPIC* (31.6), which uses the same improved backbone but follows the conventional supervised Point ICL training protocol. TETRIS pretraining also improves downstream data efficiency on ShapeNet-In-Context, especially under limited supervision.

Critical strengths
  1. The paper identifies an important evaluation issue in Point ICL. The distinction between task recognition and genuine task inference is well motivated. The paper correctly notes that changing the object domain does not necessarily test generalization to new transformation processes.

  2. PIC-Bench provides a more structured evaluation of transformation transfer. The benchmark covers sensor noise, clutter, multi-view misalignment, partial observations, and physical occlusion, with multiple subtypes and difficulty levels. This is more informative than evaluating only simple random perturbations.

  3. The self-supervised design of TETRIS is well aligned with its objective. Applying the same transformation composition to different support and query objects encourages the learned representation to capture shared transformation information rather than memorizing support geometry. The formulation is also appealing because it does not require task labels or downstream training pairs.

Critical weaknesses
  1. The “unseen task/process” claim needs to be defined more precisely.
    Several PIC-Bench degradations appear conceptually related to the atomic operators used during TETRIS pretraining, including point addition/removal, rotation, jittering, and missing geometry. It is therefore unclear whether the benchmark primarily evaluates unseen operators, unseen compositions, parameter extrapolation, or more structured realizations of previously seen operators. If the test processes are largely composed of familiar primitive transformations, the claims should be framed more carefully as compositional or structured transfer rather than broad generalization to unseen processes.

  2. The connection between PIC-Bench and real acquisition failures is not sufficiently validated.
    Many degradations are synthetically parameterized by the authors, such as range bias, scanline wobble, occlusion patterns, and multi-view pose errors. While these are plausible, the paper does not provide comparisons with real sensor artifacts, real scanning failures, or established real-world corrupted point cloud datasets. This makes it difficult to assess how representative the benchmark is of practical 3D acquisition scenarios.

  3. The method may depend strongly on the coverage of the atomic operator library.
    TETRIS is pretrained using addition, removal, rotation, scaling, and jittering, while the benchmark tasks involve related corruption factors. The current ablations mainly vary composition length, but do not study operator removal, alternative operator distributions, or test processes that are clearly outside the pretraining operator family. As a result, it is difficult to determine how much of the gain comes from learning a general task representation versus covering the key structures of the test degradations during pretraining.

Questions for authors
  1. Could the authors clarify the relationship between each PIC-Bench task and the atomic operators used during TETRIS pretraining? In particular, which aspects are genuinely unseen at test time: operator type, composition, parameter range, or structured process?

  2. How were the PIC-Bench degradation processes and severity ranges chosen? Were they calibrated using real sensor data, real scanning failures, or domain expertise? Validation on real corrupted point clouds, or at least a comparison between synthetic and real degradations, would strengthen the benchmark.

  3. How sensitive is TETRIS to the design of the atomic operator library? For example, what happens if point addition, removal, rotation/scaling, or jittering is removed during pretraining, or if different parameter distributions are used? Such experiments would help separate general task-representation learning from coverage of the benchmark degradations.

2 comments

We thank the reviewer for the constructive feedback.

1. Meaning of unseen processes. We would like to clarify that “unseen” refers to the complete degradation process. TETRIS constructs training episodes from a shared library of seven atomic operators: global and local point addition, global and local point removal, rotation, scaling, and jittering. These operators introduce uniform outliers or Gaussian clusters, remove points randomly or directionally, rotate or scale the entire point cloud, and apply independent Gaussian perturbations. For each episode, we sample the operator identities, order, and parameters online and apply the same composition to different support and query objects.

PIC-Bench uses independently defined generators based on sensor, scene, visibility, and multi-view mechanisms. These generators introduce specific observation rules and spatial dependencies, and none is used to construct the pretraining episodes. The five processes are:

  • Structured Sensor Denoising (SSD) models sensor-dependent distortions through range bias fields, depth quantization, scanline wobble, and multipath ghost returns. Range bias varies across the observed surface, quantization creates depth bands, scanline wobble introduces periodic displacement, and ghost returns form displaced surfaces. The resulting distortions depend on the sensor configuration and object geometry.

  • Scene Clutter Removal (SCR) constructs observations containing the target object and surrounding clutter. Its subtypes introduce supporting planes, fragments of other objects, points extending beyond the target boundary, or mixtures of these sources. The clutter has coherent surface structure and a spatial relationship to the target, requiring the model to recover the target geometry from the combined observation.

  • Multi-View Misalignment (MVM) generates partial scans from multiple viewpoints and fuses them after introducing pose errors. Its subtypes cover pairwise pose errors, accumulated drift across scans, a single misaligned view, and limited overlap between views. The fused observation contains inconsistent geometry determined by the individual scan poses and their visible regions.

  • Rendered Partial Completion (RPC) constructs partial observations using viewpoint-dependent visibility. Its subtypes include single-view observations, sparse multi-view coverage, additional thinning of observed points, and depth-dependent density variation. The observed geometry depends on the viewpoints, surface visibility, and sampling pattern, while the reconstruction target is the complete object.

  • Physical Occlusion Recovery (POR) places foreground objects or planar screens between the sensor and the target and computes visibility in the combined scene. The input contains both visible target regions and foreground interference. Its subtypes vary the occluder geometry, number, and proximity to the target, producing different patterns of missing geometry and obstruction.

The distinction is therefore between online compositions of atomic point-set operations and structured processes governed by sensor, scene, visibility, and multi-view rules. Elementary corruption factors and numerical parameter ranges may overlap; PIC-Bench evaluates whether learning from the former transfers to the latter.

We will add precise definitions of all five task generators and detailed parameter settings to the appendix, including subtype-specific generation rules, visibility and occlusion procedures, and numerical ranges for each difficulty level. We will also clarify their distinction from the atomic pretraining operators in the main text.

2. Acquisition mechanisms and severity ranges. PIC-Bench uses synthetic models motivated by sensor errors, scene interference, partial visibility, and multi-view acquisition. We define five severity levels on normalized point clouds by increasing displacement, clutter, occlusion, and pose errors, or decreasing visible coverage. These ranges provide controlled variation in degradation severity and were not calibrated against real sensor measurements. We will state this design rationale alongside the detailed generator specifications in the appendix.

3. Sensitivity to the operator library. Following the reviewer’s suggestion, we added four operator-removal ablations on ShapeNet. All variants use the same architecture and training settings, including 300 epochs, with only the available operators changed. Removing Add or Drop excludes both global and local variants. We evaluate all models on the complete 10,000-episode PIC-Bench test set.

Operator-library ablation (CD ↓).

SettingSSDSCRMVMRPCPORAvg.
Full library5.439.0910.356.7618.099.94
w/o Add6.1822.6612.075.9521.9413.76
w/o Drop5.559.1211.0127.8527.8216.27
w/o Rotate/Scale5.6911.1511.875.9015.279.98
w/o Jitter6.1110.3110.656.7817.5110.27

The full operator library achieves the lowest average CD, while reduced libraries show different performance profiles across tasks. These results suggest that the atomic operators provide complementary training signals. Their joint use broadens the range of transformation compositions encountered during pretraining and supports transfer across the diverse structured processes in PIC-Bench.

Reviewer_UyDNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether point-cloud in-context learning transfers to degradation processes not used during training. It introduces PIC-Bench, covering five structured restoration tasks, and TETRIS, a self-supervised framework that applies shared compositions of atomic operators to different clean objects. An improved DeformPIC backbone learns to use a support pair to restore a query point cloud. Experiments report gains on PIC-Bench and in low-label downstream fine-tuning, with same-backbone comparisons, composition ablations, and support-utilization tests examining the contributions of the training procedure and support information.

Critical strengths
  1. PIC-Bench extends Point ICL evaluation to structured degradation processes that are not used as training tasks. Its five restoration tasks test transfer beyond the familiar task processes used in existing protocols. This is a useful extension of evaluations that change the object distribution while retaining the same task definitions.

  2. The training framework is tested with relevant controls. TETRIS reduces CD from 31.6 to 9.9 when both it and DeformPIC* use the same backbone and train on ShapeNet (Table 1). The separately trained query-only baseline matches the training data, atomic compositions, predictor, and optimization schedule. Using correct support improves CD from 11.8 to 8.7 and F-score from 37.2% to 41.2% over this baseline (Table 4), providing direct evidence that support information improves restoration.

Critical weaknesses

1.The evaluation does not separate changes in the object distribution from changes in the degradation process. The published baselines are trained on ShapeNet, whereas PIC-Bench uses OpenShape-LVIS objects (Section 3.3, Table 1). The ShapeNet-trained TETRIS comparison controls the training source between methods and supports its relative advantage, but does not isolate these two sources of test-time shift. Baseline failure therefore cannot be attributed specifically to unfamiliar degradation processes. Evaluating familiar and PIC-Bench degradations on the same held-out objects would make the task-transfer conclusion more precise.

2.The role of support is not examined separately across tasks with different sources of ambiguity. Support and query come from different objects and share degradation parameters (Section 3.1). A support pair can provide information about the degradation, but cannot uniquely determine the query’s unobserved geometry in RPC and POR. Table 4 reports only aggregate support gains, leaving it unclear whether conditioning mainly helps correct observable distortions or also improves reconstruction in the ambiguous settings. Correct-support and query-only results for each task would clarify where the task representation helps, including tasks with substantial missing geometry.

Questions for authors

1.How do the ShapeNet-trained baselines perform on familiar training-task degradations applied to the same OpenShape-LVIS test objects? Comparing these results with PIC-Bench would help separate the effects of the object distribution and the degradation process.

2.Can you report the correct-support and query-only results in Table 4 separately for all five tasks, particularly RPC and POR? This breakdown would show whether the overall support benefit is concentrated in particular restoration settings.

3.For reproducibility, please specify the numerical difficulty ranges, visibility and occlusion rules, input resampling, and F-score threshold for PIC-Bench. Please also clarify which checkpoint produced the ShapeNet result in Table 1: Section 5.2 gives 300 training epochs, while Appendix B.3 states that self-supervised models are evaluated at epoch 100.1,

1 comment

We thank the reviewer for the constructive feedback.

1. Object distribution and degradation process. Following the reviewer’s suggestion, we evaluated original reconstruction and denoising alongside RPC and SSD on both held-out ShapeNet and LVIS objects. All four models use fixed ShapeNet-trained checkpoints. Within each test dataset, each comparison applies the original and PIC-Bench processes to the same 2,000 support–query object pairs.

Comparison of original and PIC-Bench processes (CD ↓).

ModelTraining dataTest objectsOriginal ReconstructionRPCOriginal DenoisingSSD
PIC-CatShapeNetShapeNet4.8846.135.9925.65
PIC-CatShapeNetLVIS6.3053.696.9029.71
PIC-SepShapeNetShapeNet4.40128.197.52106.46
PIC-SepShapeNetLVIS5.72131.339.55106.95
DeformPICShapeNetShapeNet2.71144.773.50104.00
DeformPICShapeNetLVIS3.50137.684.0890.85
TETRISShapeNetShapeNet5.503.463.853.90
TETRISShapeNetLVIS10.056.766.515.43

The baselines perform substantially better on original processes on both object datasets. Their errors on RPC and SSD remain high even on ShapeNet, showing that the LVIS object shift alone does not explain their PIC-Bench performance. TETRIS achieves the lowest CD on RPC and SSD in both settings.

2. Per-task support results. We report the per-task results for the OpenShape-pretrained models in Table 4. We additionally evaluate 8-shot inference by averaging task tokens from eight support pairs using the same frozen TETRIS checkpoint.

Per-task results (CD ↓).

TaskQuery-onlyTETRIS, 1-shotTETRIS, 8-shot
SSD7.385.565.17
SCR18.127.957.51
MVM11.988.777.46
RPC5.784.534.46
POR11.3711.1110.34
Avg.11.818.737.94

Avg. is computed over the complete 10,000-episode test split.

3. Reproducibility and checkpoint clarification. We will add the numerical parameter ranges for every subtype and difficulty level, together with the full generation rules, to the appendix. Visibility uses projected z-buffering, with visible subsets adjusted to the prescribed ratios; POR computes visibility jointly for the target and foreground occluders. Inputs are randomly resampled to 1,024 points, with replacement only when fewer points are available; predictions are reduced by farthest-point sampling. F-score uses a threshold of 0.001 on squared nearest-neighbor distances.

The ShapeNet result in Table 1 uses the final epoch-300 checkpoint, while OpenShape pretraining uses the final epoch-100 checkpoint. We will correct the checkpoint description in the appendix.

Reviewer_QYWBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether Point In-Context Learning (Point ICL) models infer transformations from support examples rather than merely recognize predefined task families. It introduces PIC-Bench, comprising five structured restoration tasks whose complete degradation processes are not used to construct TETRIS training episodes. TETRIS applies the same sampled operator composition, including its order and parameters, to different clean objects, while sampling stochastic effects independently. One object's corrupted-clean pair then guides restoration of the other. The model builds on DeformPIC, with changes to support encoding and conditioning. Experiments report improved performance on PIC-Bench and more effective downstream fine-tuning with limited paired data.

Critical strengths
  • PIC-Bench targets the evaluation gap between generalization to different degradation processes, rather than only to different objects or domains. The distinction between recognizing a familiar task and inferring its demonstrated transformation is worth examining, and the benchmark gives a concrete setting in which to study it.

  • The ShapeNet comparison in Table 1 provides an important control. TETRIS achieves 9.9 CD versus 31.6 for DeformPIC* with the same backbone and source dataset, so neither the architectural changes nor access to OpenShape can alone explain the improvement. The low-data fine-tuning gains in Figure 3 provide further evidence that the pre-training is useful beyond PIC-Bench.

Critical weaknesses
  • The support experiments leave the task-inference interpretation unresolved. The separately trained query-only model reaches 11.8 CD, compared with 26.6 for Identity and 8.7 for TETRIS (Tables 1 and 4). Most of the improvement over Identity is thus attainable without support; this comparison does not decompose the contributions within TETRIS. At the same time, TETRIS with correct support has about 26% lower CD than the matched query-only model. The unresolved question is whether this additional benefit reflects recognition of the corruption type or inference of episode-specific transformation information. The paper states that support pairs are shuffled across episodes, but does not specify whether task or subtype is preserved.

  • The train-test process relationship is not characterized precisely enough. The training and test corruptions are related at the level of extra points, missing regions, pose changes, and perturbations, but their implementations differ. Within each training and test episode, the support and query are different objects that share episode-level transformation parameters (Sections 3.1 and 4.2). The evidence therefore needs to be interpreted within this common episodic setup. My concern is not that the complete test generators were used in training, but that the process-level separation is difficult to assess from the available definitions.

Questions for authors
  • Does the shuffling in Table 4 preserve task or subtype? Within a fixed subtype and difficulty level, how much does matching the support's parameters to the query matter?
1 comment

We thank the reviewer for the constructive feedback.

1. Support shuffling and parameter matching. We would like to clarify that Table 4 uses global support shuffling without preserving task or subtype. We added a comparison on 500 episodes across all five tasks using the same frozen TETRIS checkpoint. We keep each query and clean support object fixed and resample support parameters within the same task, subtype, and difficulty level. Matching parameters reduces average CD from 10.47 to 9.26; after additionally matching degradation severity, from 10.26 to 9.28.

Support-parameter matching on PIC-Bench.

Support parametersCD ↓CD ↓ (severity matched)
Matched9.269.28
Mismatched10.4710.26

We further constructed a controlled two-view MVM setting, holding the fused scans fixed while varying the reconstruction frame specified by support. Across 200 queries, each under two support conditions, TETRIS selects the demonstrated frame with 90.75% accuracy, compared with 50% for query-only and Identity. Matching and swapped support give 5.88 and 14.90 CD, respectively, against the same requested target.

Reference-selection accuracy is the fraction of predictions closer in CD to the support-specified target than to the alternative; an unchanged prediction across the two support conditions therefore scores 50%.

Controlled MVM diagnostic.

MethodMatched CD ↓Swapped CD ↓Acc. (%) ↑
TETRIS5.8814.9090.75
Query-only13.8013.8050.00
Identity6.476.4750.00

With the query and corruption type unchanged, these results show that support conveys episode-specific transformation information.

2. Training–test relationship. Both use different support and query objects with shared episode-level parameters. TETRIS trains on online atomic compositions; PIC-Bench evaluates independently defined structured generators. Specifically, training adds uniform outliers or Gaussian clusters, while scene clutter removal introduces planes or fragments of other objects. Independent Gaussian jitter differs from range-dependent bias and scanline distortions; whole-cloud rotation differs from per-view pose errors; random or directional point removal differs from visibility- and occlusion-driven missing regions. We will clarify these distinctions in the revision.

Reviewer_Fq5oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies point-cloud in-context learning under transformation processes not used during training. It introduces PIC-Bench, a benchmark containing five structured restoration tasks, and proposes TETRIS, which constructs self-supervised training episodes by applying the same composition of atomic point-set operators to different support and query objects. The method builds on DeformPIC with modifications to support encoding and query conditioning. Experiments demonstrate improved transfer on PIC-Bench and better downstream adaptation under limited supervision.

Critical strengths
  1. A useful evaluation perspective. Distinguishing generalization to new objects or domains from transfer to previously unused corruption processes is meaningful. PIC-Bench provides a concrete evaluation of the latter across several structured restoration settings and exposes limitations of existing Point ICL models.

  2. Substantial empirical improvements with relevant controls. Using the same backbone and ShapeNet training-shape source, TETRIS reduces overall PIC-Bench CD from 31.6 to 9.9. The paper also includes composition-length and support-utilization ablations, and reports substantial improvements under limited downstream supervision. These results provide evidence that the proposed training protocol is useful.

Critical weaknesses
  1. Insufficient isolation of the methodological contribution. DeformPIC already provides task-token extraction and AdaLN conditioning, while STL explores cross-instance transformation representations. The DeformPIC* comparison supports the proposed training protocol, but does not isolate the effects of independent support sampling, multiple task tokens, or cross-attention. Component ablations and training the original DeformPIC with the proposed protocol would clarify these contributions.

  2. Unclear architectural presentation. Figure 1 provides a conceptual overview, leaving readers to reconstruct the support-to-query computation from Section 4.3 and Appendix B. A dedicated architecture diagram should show the task-token extraction and conditioning pathways, explicitly distinguishing inherited components from the modifications to DeformPIC.

Questions for authors
  1. How much do the architectural modifications contribute beyond the training protocol? Please compare the original DeformPIC backbone under the proposed training protocol and provide targeted ablations, particularly for multiple task tokens and cross-attention.

  2. Could the authors clarify the complete support-to-query computation? An architecture diagram showing task-token extraction, cross-attention, and AdaLN conditioning, with inherited and newly introduced components clearly distinguished, would help assess the changes relative to DeformPIC.

1 comment

We thank the reviewer for the constructive comments. We have added architecture ablations and diagrams to clarify our design.

1. Architectural contributions beyond the training protocol.

Following the reviewer's suggestion, we conducted progressive ablations under the same TETRIS training protocol, using identical ShapeNet training shapes, optimization settings, and training budgets. All results are evaluated on PIC-Bench using the epoch-300 checkpoints.

ArchitectureTask tokensCross-attentionIndependent samplingCD ↓
Full TETRIS8✓✓9.94
Single task token1✓✓10.39
Single task token without cross-attention1—✓11.01
DeformPIC1——11.42

Reducing the task-token count, removing cross-attention, and replacing independent sampling with coupled sampling progressively increase CD from 9.94 → 10.39 → 11.01 → 11.42. These results demonstrate the additional contribution of each component under a fixed training protocol.

2. Relation to STL.

We would like to clarify that STL and TETRIS share the idea of learning transformation representations across instances, but differ in their objectives. STL applies the inferred transformation in latent feature space to improve image representations for downstream classification and detection. TETRIS uses the support-derived representation to predict the transformed point cloud of a query object.

For supervision, STL uses image augmentations to learn latent equivariance and align transformation representations across instances. TETRIS uses online compositions of atomic point-set operators to generate support–query restoration episodes and trains through query reconstruction. We further introduce PIC-Bench, a benchmark specifically designed for Point ICL, to evaluate support-conditioned generalization to structured task processes absent from pretraining.

3. Architectural presentation.

We would like to clarify that our core contributions are PIC-Bench and the TETRIS learning algorithm. We apologize for not describing the architecture details clearly enough. To address this issue, we have added a new section to the appendix that provides a detailed description of the architecture. Our architectural changes build on DeformPIC by introducing independent support patch sampling, multiple task tokens, and cross-attention injection. The reviewer’s suggestion highlights that the missing architecture overview made these changes and their role in TETRIS harder to follow. We have therefore added an overview diagram to the main text, with a detailed diagram in the appendix, clearly distinguishing inherited components from our modifications.

Reviewer_HmL3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed PIC-Bench and TETRIS to study point-cloud in-context learning. The method focused on composing atomic operators for different objects and applying such a mechanism as a self-supervised learning framework to point-cloud in-context learning. In the experiment, the author shows convincing performance on multi-task point cloud learning.

Critical strengths
  1. The PIC-Bench provides a comprehensive evaluation protocol by testing structured corruption processes beyond the predefined tasks used to train the comparison methods.

  2. The self-supervised training formulation is aligned with the point-cloud ICL motivation. Applying the composition to different objects across different tasks creates a learning signal for transferring transformation-related information across object geometries.

  3. The empirical results show significant improvements: TETRIS CD of 9.9, compared with 31.6 for DeformPIC; with correct support, CD improves from 11.8 for the matched query-only model to 8.7; and with low-label adaptation using 1% of the downstream labeled data, pretraining reduces CD from 9.82 to 4.94.

Critical weaknesses
  1. In Table 4, it is shown that support helps, but it does not distinguish between inferring a new transformation and selecting or calibrating a learned restoration behavior.

  2. Figure 3(c) shows t-SNE clusters for different transformations, but does not quantify how the representation changes across support objects under the same transformation. The extent to which it separates transformation information from object geometry remains unclear.

  3. Figure 3(b) compares TETRIS pretraining with random initialization. This establishes a pretraining benefit, but does not separate the contribution of task conditioning from the geometric features learned through restoration pretraining.

Questions for authors
  1. How can we distinguish task learning from task recognition in this setting?
  2. For a fixed transformation and query, how do the task representation and prediction change when the support object changes?
  3. How does the query-only pretrained model from Table 4 perform under the same 1% and 5% fine-tuning protocols?
1 comment

We thank the reviewer for the insightful comments. We have clarified the evaluation protocol and conducted additional experiments on support-object variation and low-data fine-tuning, as detailed below.

1. Task learning vs. task recognition.

We would like to clarify that task recognition selects or calibrates a familiar restoration behavior, while task learning infers task-specific transformation information from the support demonstration to guide the query transformation. We examine this distinction through two complementary aspects:

  • Task novelty. PIC-Bench defines a task by its complete degradation–restoration process. Broad labels such as “denoising” or “completion” describe restoration objectives but do not specify these processes. TETRIS is pretrained only on atomic compositions, and none of PIC-Bench’s five structured task generators is used during pretraining. The evaluation therefore tests generalization to unseen task processes, even when the restoration objectives are familiar.
  • Support utilization. Table 4 examines whether demonstrations contribute to generalization in this unseen-task setting. Correct support reduces CD from 11.8 for the matched query-only model to 8.7, showing that support provides useful information beyond query geometry alone.

Together, these results support our claim that TETRIS uses support demonstrations to generalize to unseen task processes. We will clarify the complementary roles of task novelty and support utilization in the revision.

2. Effect of changing the support object.

Following the reviewer's suggestion, we conducted a quantitative comparison across 20 object categories. The table reports the average cosine similarity between global task vectors; ✓ indicates that the corresponding attribute is shared.

Same transformationSame categorySame instanceCosine similarity
✓✓0.734
✓0.588
✓✓0.169

Task vectors are more similar when the transformation is shared, even across object categories, than when different transformations are applied to the same object. Sharing the object category further increases similarity. These results quantitatively support the transformation-related structure illustrated in Fig. 3(c).

For predictions, we fixed the query and transformation while changing the support object. Same-category support reduces average recovery CD by 18.9%, from 7.199 to 5.840.

3. Task conditioning vs. geometric pretraining.

Following the reviewer's suggestion, we added the query-only-pretrained model from Table 4 to the same 1% and 5% fine-tuning protocols. During fine-tuning, this control uses the same support-conditioned architecture, labeled subsets, and optimization settings as TETRIS.

Initialization1% data: CD ↓5% data: CD ↓
Random9.86.7
Query-only pretraining8.16.4
TETRIS pretraining4.93.3

Query-only pretraining yields only marginal improvements over random initialization, suggesting that geometric restoration pretraining alone offers limited benefits. By contrast, TETRIS further reduces the average CD by 39.0% and 48.3% under 1% and 5% supervision, respectively, relative to this control. This comparison indicates that the main gains arise from task-conditioned pretraining, which substantially improves downstream adaptation beyond the benefits of geometric restoration pretraining alone.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your comments and suggestions. We have posted our responses and revised manuscript, and would like to briefly summarize the main clarifications and new results.

We clarified what we mean by “unseen” task processes. PIC-Bench retains familiar objectives such as denoising and completion, while changing the processes that generate the observations. To separate this change from differences in object distribution, we compared the original and PIC-Bench processes on the same object pairs within ShapeNet and LVIS. The baselines still struggle with the PIC-Bench processes on ShapeNet, suggesting that the LVIS object shift alone does not explain the performance gap.

We also added a comparison with query-only pretraining under the same low-data fine-tuning settings. With 1% and 5% labeled data, TETRIS achieves CD values of 4.9 and 3.3, compared with 8.1 and 6.4 for query-only pretraining. Our individual responses further examine how task representations and predictions change with the support object and the transformation demonstrated by support.

To make the method easier to follow, we added architecture diagrams and progressive ablations of the task tokens, cross-attention, and support patch sampling. Under the same TETRIS training protocol, the full architecture achieves CD 9.94, compared with 11.42 for the DeformPIC architecture. We also clarified how TETRIS relates to STL.

The new operator-removal experiments show how the training library affects different tasks. Removing Add particularly hurts clutter removal, while removing Drop hurts completion and occlusion recovery. The full library performs best on average, although some reduced libraries perform better on individual tasks.

We hope these updates help address your questions. Please let us know if any points remain unclear or would benefit from further discussion. Thank you again for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.