acceptodds
Under review as a conference paper at ICLR 2027

Pairformer Is an Effective Relational Embodied Learner

Abstract

Transformers thrive on large-scale data, yet robot experience remains costly and sparsely covers physical interactions. We investigate a relational inductive bias for embodied learning: maintaining and composing vector-valued representations of token pairs. We study Pairformer from AlphaFold 3, whose persistent pair states and triangle updates guide token attention while retaining a Transformer-compatible token interface. Diagnostic experiments show that Pairformer retains graph relations for subsequent queries and predicts contact-driven motion more accurately, with attention more closely aligned to collisions in a visual probe. We then test this computation in three embodied components: (i) observation encoders for diffusion policies, (ii) latent dynamics models for model-predictive control, and (iii) pretrained action predictors within . On seven simulated manipulation tasks with 200 demonstrations per task, the encoder and dynamics adaptations improve mean success by 6.9 and 2.85 percentage points over their Transformer baselines. The action-predictor adaptation improves atomic-seen success by 4.23 points, with smaller gains on composite tasks. Together, these results support explicit relational computation as an effective architectural complement to data scaling and pretraining for embodied learning and control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Pairformer, Relational Inductive Bias, Robot Learning, Embodied AI, World Models, Model Predictive Control, Vision-Language-Action Models, Diffusion Policies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_heVROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts Pairformer to diffusion-policy observation encoders, MPC latent dynamics models, and the action expert of π0.5. By explicitly maintaining and updating pairwise representations, it aims to improve relational reasoning in embodied learning. Simulation results show consistent gains on low-data manipulation and atomic interaction tasks.

Critical strengths
  1. Pairformer is evaluated in diagnostic tasks, behavior cloning, model-based planning, and pretrained VLA adaptation rather than in a single setting.
  2. It improves all seven BC tasks and all seven MPC tasks over the corresponding Transformer baselines, with particularly meaningful gains on contact-rich tasks.
  3. The paper documents data splits, seeds, model sizes, evaluation episodes, and computational costs unusually well.
Critical weaknesses
  1. The comparisons are not parameter- or compute-matched,Pairformer also changes geometric pair inputs, gating, and transition operators, so the source of improvement is unclear.
  2. Attention–contact alignment is measured on only one post-hoc-selected billiards layout and therefore provides limited explanatory evidence.
  3. Pairformer has substantial computational cost: MPC world-model training rises from 122.0 to 371.6 GPU-hours, and pair operations have unfavorable scaling.
  4. All robot experiments are conducted in simulation, and improvements on compositional VLA tasks are small.
Questions for authors
  1. Pairformer has more parameters than the Transformer baseline. How can the authors rule out that the gains mainly result from increased model capacity? Please include a parameter-matched Transformer baseline
  2. Why does Pairformer provide almost no improvement on compositional tasks? Can you provide a failure-case analysis?
  3. The attention–collision correlation is measured on only one billiards layout. Can the authors evaluate this correlation across more layouts to support the interpretability claim?
  4. Can the authors provide real-robot experiments to verify whether the improvements observed in simulation transfer to physical manipulation settings?
1 comment

Response to Reviewer heVR

W1 / Q1. Model capacity and architectural attribution. Pairformer improves the original seven-task BC result from 61.1% to 68.0% (+6.90 pp) and MPC from 64.79% to 67.64% (+2.85 pp). In the focused Ex1/Ex4 results, Pairformer lowers Billiards RMSE at all three horizons; BC/MPC gains are larger in Ex1 than Ex4.

SettingModelBC (%)MPC (%)RMSE@16/32/64 (cm)Selection (%)
Ex1Transformer35.00±2.5035.00±1.800.230/0.750/2.27081.48±1.13
Ex1Pairformer w/o tri.37.00±1.0036.00±1.320.203/0.623/1.82783.45±0.95
Ex1Pairformer39.00±2.7838.00±2.650.190/0.543/1.50785.15±0.88
Ex4Transformer37.50±2.2937.50±1.800.213/0.657/1.88083.50±1.03
Ex4Pairformer w/o tri.39.00±1.8038.83±1.890.190/0.567/1.60084.85±0.97
Ex4Pairformer39.00±2.7838.00±2.650.190/0.543/1.50785.15±0.88

Mean±sample SD over three seeds; BC/MPC use 200 episodes/seed and Billiards selection uses 2,000. The original complete RGB-DP configurations contain 74.138M Transformer and 75.643M Pairformer parameters (+2.03%). The original MPC world models contain 3.594M and 5.098M parameters (+41.86%). The no-triangle ablation retains pair states but removes triangle operations without width compensation; it is not a capacity-matched pair-local control. Ex1 and Ex4 are the parameter- and compute-budget controls, respectively. Across the original seven tasks, Pairformer improves all seven BC and all seven MPC task means (two-sided exact sign test p=.016 for each).

W2 / Q3. Attention–contact alignment. Attention–contact correlations are 0.531 for Transformer and 0.747 for Pairformer on one post-hoc-selected held-out layout. This is a single-layout diagnostic. In the primary 300k-trajectory BilliardsRelay evaluation, RMSE at horizons 16/32/64 decreases from 0.270/0.890/2.960 cm to 0.180/0.510/1.380 cm, and selection success increases from 79.27% to 85.45%. These results support the prediction comparison, not cross-layout attention alignment.

W3. Computational cost. A Pairformer world-model step takes 4.907 ms versus 0.870 ms for Transformer (5.64×); training takes 371.6 versus 122.0 GPU-hours (3.05×). Mean full decision-cycle latency rises from 32.27 s to 35.44 s (+9.8%; P95: 32.38 s to 35.61 s). The current configuration does not demonstrate high-frequency control.

W4 / Q2. Composite VLA performance. The gain is concentrated on atomic-seen tasks: 37.88% to 42.11% (+4.23 pp; 18 tasks). Composite-seen changes from 10.00% to 10.25%, and composite-unseen from 1.00% to 1.25%. Both composite groups have near-floor success rates, so these +0.25 pp differences provide little useful evidence about compositional generalization. The pair branch receives noisy action-token projections and relative action-time features; visual-language context enters through the retained single stream. Evaluation uses three training seeds and 50 episodes/task/seed.

W4 / Q4. Simulation-to-real transfer. All manipulation experiments are simulated; physical-robot transfer has not been evaluated.

Reviewer_6gCQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether an explicit relational inductive bias—persistent, vector-valued pair states composed via AF3-style triangle operations and used to bias token attention (Pairformer)—improves embodied learning with limited robot data. The authors adapt Pairformer to three components: observation encoders for diffusion policies, latent dynamics for MPC, and a parameter-efficient pair-bias adapter for a pretrained VLA action expert; they also design two targeted diagnostics (directed reachability reuse and billiards contact prediction) to probe representational and physical reasoning benefits. Across seven simulated manipulation tasks and a RoboCasa365 VLA benchmark, Pairformer variants consistently improve success rates over strong Transformer baselines, with diagnostic evidence that pair-state composition preserves reusable graph information and better aligns attention with contact events.

Critical strengths
  1. A coherent adaptation of persistent relational computation to embodied components. The paper does more than propose a new block in isolation: it shows how pair states can be initialized from task-specific token and geometric inputs and then inserted into observation encoding, latent prediction, and action generation while retaining surrounding interfaces. This makes the architectural idea easy to test and potentially reusable.

  2. Useful mechanism-oriented diagnostics. The delayed-query reachability task directly tests whether relations can be reused after encoding, and the billiards experiment connects attention routing to contact events and long-horizon prediction. These probes make the claimed relational bias more interpretable than a task-success comparison alone.

Critical weaknesses
  1. The embodied comparisons do not isolate the effect of Pairformer sufficiently. In the diffusion-policy encoder, the Pairformer configuration also changes the pair inputs (including geometric information), gating, and transition details, and it has a different parameter count. The paper itself states that the comparison evaluates complete setting-specific adaptations. Thus the +6.9-point gain cannot be attributed specifically to persistent pair storage or triangle computation. The MPC triangle ablation also changes capacity and operator structure, while there is no parameter/FLOP-matched alternative. Stronger controls are needed before concluding that the relational mechanism itself is responsible for the improvements.

  2. The practical and conceptual evidence for general relational understanding is limited. All robot experiments are in simulation. The pi0.5 pair branch is conditioned on noisy action tokens and relative action indices, while the visual-language prefix is handled by the retained single stream; therefore this experiment primarily tests action-sequence coordination rather than explicit object-contact reasoning. Its gains are 4.23 points on atomic-seen tasks but only 0.25 points on each composite group. At the same time, Pairformer incurs quadratic pair-state storage and substantial planning cost: the reported MPC cycle increases from 32.27 s to 35.44 s, world-model training from 122.0 to 371.6 GPU-hours, and the world-model step from 0.870 to 4.907 ms. This weakens the claim that the approach is broadly effective for real-time embodied control.

Questions for authors
  1. Can the authors provide parameter-, FLOP-, and activation-memory-matched controls for the observation encoder and MPC world model? In particular, what is the gain after matching the extra geometric inputs, gating, transition capacity, and total depth/width?

  2. In the pi0.5 adapter, does the pair branch receive any information derived from the visual-language prefix at any layer, or is all scene conditioning mediated indirectly through the single stream? If the latter, what evidence supports interpreting the result as relational scene reasoning rather than action-chunk regularization?

  3. Are the reported task-level gains statistically significant across independent seeds and episodes? Please report confidence intervals or per-task paired tests, especially for the small composite-task gains and for the triangle ablation.

  4. Can the authors report scaling with token count and a realistic control frequency, including peak memory and latency for the observation-encoder setting? The current MPC timing suggests that full pair computation may be unsuitable for high-frequency control.

1 comment

Response to Reviewer 6gCQ

W1. Attribution of embodied gains. In the focused Ex1/Ex4 results, Pairformer improves BC/MPC over Transformer by +4.00/+3.00 pp in Ex1 and +1.50/+0.50 pp in Ex4, and lowers Billiards RMSE at all three horizons.

SettingModelBC (%)MPC (%)RMSE@16/32/64 (cm)Selection (%)
Ex1Transformer35.00±2.5035.00±1.800.230/0.750/2.27081.48±1.13
Ex1Pairformer w/o tri.37.00±1.0036.00±1.320.203/0.623/1.82783.45±0.95
Ex1Pairformer39.00±2.7838.00±2.650.190/0.543/1.50785.15±0.88
Ex4Transformer37.50±2.2937.50±1.800.213/0.657/1.88083.50±1.03
Ex4Pairformer w/o tri.39.00±1.8038.83±1.890.190/0.567/1.60084.85±0.97
Ex4Pairformer39.00±2.7838.00±2.650.190/0.543/1.50785.15±0.88

Mean±sample SD over three seeds; BC/MPC use 200 episodes/seed and Billiards selection uses 2,000. The original complete RGB-DP configurations contain 74.138M Transformer and 75.643M Pairformer parameters (+2.03%); the original MPC world models contain 3.594M and 5.098M parameters (+41.86%). The no-triangle ablation retains pair states but removes triangle operations without width compensation, so it is not a capacity-matched pair-local control.

W2. Scope and control cost. The π0.5 adapter performs pairwise computation over action positions. Its pair state is initialized from noisy action-token projections and relative action-time features, persists across 18 action-expert layers, and supplies eight query-head biases. The visual-language prefix conditions the retained single stream. The pair branch does not directly receive the visual-language prefix. For MPC, a Pairformer world-model step takes 4.907 ms versus 0.870 ms for Transformer (5.64×); training takes 371.6 versus 122.0 GPU-hours. Mean full decision-cycle latency is 35.44 s versus 32.27 s. This configuration does not demonstrate high-frequency control.

Q1. Resource matching. Ex1 is the parameter-budget control and Ex4 the compute-budget control. The parameter counts above are for the original complete RGB-DP policy and original MPC world models, not these controls. The available Ex1/Ex4 outcome records do not contain control-specific parameter counts, FLOPs/MACs, or peak activation memory, nor do they establish matching of geometric inputs, gating, transition capacity, depth, and width. We report the measured outcomes without claiming that all resources are matched.

Q2. VLA information path. The action tensor is [B,50,32] and pair tensor [B,50,50,64]. Pair states are initialized from noisy action-token projections and relative action-time features, persist through the 18 action-expert layers, and generate eight query-head biases over action positions. The visual-language prefix KV cache remains in the single stream; action queries retain joint normalization over prefix and action keys. The zero-initialized pair-bias output preserves the pretrained policy output at initialization. The adapter adds 2,681,024 parameters (about 0.080% of the full policy). Thus, the experiment evaluates action-position coordination in an observation-conditioned policy.

Q3. Statistical evidence. Across the original seven tasks, Pairformer exceeds Transformer on all seven BC and all seven MPC task means; two-sided exact sign tests over tasks give p=.016 for each. These are task-level tests, not seed-level paired tests. In Ex1/Ex4, BC/MPC differences are +4.00/+3.00 pp and +1.50/+0.50 pp, respectively; all four unpaired three-seed Welch 95% CIs include zero. For Billiards RMSE@64, Ex1 reduction is 0.763 cm (95% CI [0.300,1.226], p=.012), while Ex4 reduction is 0.373 cm ([−0.018,0.765], p=.057). Selection gains are +3.67 pp in Ex1 ([1.32,6.02], p=.013) and +1.65 pp in Ex4 ([−0.53,3.83], p=.103).

Q4. Token scaling and frequency. The MPC world model uses 65 tokens and a 65×65×32 pair state; the observation encoder uses 152 tokens. Pair-state storage scales quadratically with token count, and triangle updates involve cubic token interactions. The available records do not contain a 64/152/256-token latency and peak-memory sweep. At the evaluated MPC setting, the Pairformer world-model step is 4.907 ms versus 0.870 ms for Transformer, and the mean full decision cycle is 35.44 s versus 32.27 s.

Reviewer_SdpjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper adapts the Pairformer computation from AlphaFold 3 to embodied learning. Pair representations persist across layers, undergo triangle updates, and influence token attention through learned biases. The study includes graph and billiards diagnostics and three embodied applications: observation encoders for diffusion policies, latent dynamics for model-predictive control (MPC), and the action expert of the pretrained π0.5 vision-language-action policy. Experiments evaluate simulated manipulation and action generation, with a triangle-removal ablation and computational-cost measurements.

Critical strengths

The adaptation of the pretrained action expert is a concrete integration design: it retains the original single stream and adds an action-pair bias whose zero-initialized output preserves the initial policy output. The branch adds 2.681M parameters, about 0.080% of the full policy. The authors also distinguish the atomic-task gain from the much smaller composite-task gains and acknowledge the computational and simulation limits. As supporting strengths, the study applies the architecture at three different embodied interfaces, and the dynamics comparison holds the embedding, behavior prior, Value ensemble, and controller fixed while including a control that removes triangle updates.

Critical weaknesses

1. The benefit of relational composition is not isolated from other architectural changes. The world model grows from 3.59M to 5.10M parameters, about 41.9%, for a 2.85-point gain over Transformer dynamics (Tables 4 and 9). The observation encoder also changes geometry inputs, gating, and transition activation. Removing triangle updates reduces capacity and compute without compensating widths. The common planner and the authors' acknowledgment of these differences are useful, but the results establish the benefit of the complete configurations more clearly than that of relational composition itself. Parameter- and compute-matched Transformer and pair-local controls would help resolve this central attribution issue.

2. The MPC improvement comes at a substantial computational cost. Mean success rises from 64.79% to 67.64%, while world-model training increases from 122.0 to 371.6 GPU-hours and a model step from 0.870 to 4.907 ms. Full decision cycles take 32.27 and 35.44 seconds on one RTX 4090 in FP32 (Table 8). Total cycle time includes shared planning components, so the 9.8% cycle increase alone is an incomplete measure of the added model cost. These absolute latencies limit the practical control benefit of the evaluated configuration. A comparison at matched latency or compute would make the performance-cost tradeoff more convincing.

Questions for authors
  1. Evaluation protocol and result provenance. Could you provide success counts and episode denominators for each task/model/seed in Table 3, identifying which cells use the 100- or 200-episode protocol? Please connect Tables 3 and 4 to their reset banks, selected checkpoints, and dataset versions, including the rebuilt PegInsertionSide split. This would clarify the half-percentage entries and differing BC averages without assuming a numerical error.

  2. Uncertainty of small gains. Could you report per-seed outcomes and paired intervals for the main comparisons, especially the 2.85-point MPC gain and 0.25-point composite-task gains? The appendices describe statistical procedures, but the resulting intervals are not shown.

  3. Representation adaptation versus planning. Could you report the adapted-BC development control and clarify whether it was also evaluated on Table 4's independent reset bank? This would help interpret the roles of representation adaptation and planning in the 5.43-point gain over direct BC. Please also clarify which calibration diagnostics are available for the ensemble penalty, currently described as a ranking heuristic.

  4. Pretrained-policy generalization. Can task-level outcomes or a failure breakdown explain why the π0.5 improvement concentrates on atomic-seen tasks while both composite groups improve by only 0.25 points? This would clarify whether the benefit concerns execution quality or task composition.

  5. Contact diagnostic coverage. Are aggregate D4 results or contact-alignment statistics across multiple held-out layouts available? The current attention correlation uses one layout, and Figure 4 is selected post hoc. If broader evidence is unavailable, these figures should remain illustrative diagnostics.

  6. Scope of the data-efficiency claim. Which reported evidence supports complementarity with data scaling beyond the fixed 200-demonstration ManiSkill comparison? Please distinguish conclusions supported by the current simulation and interface sizes from hypotheses about data efficiency, larger scenes, and physical robots. This is a request to clarify the claim's scope; additional budget, scaling, or real-robot studies can remain future work.

1 comment

W1. Attribution of relational-composition gains. Pairformer improves the original seven-task BC result from 61.1% to 68.0% (+6.90 pp) and MPC from 64.79% to 67.64% (+2.85 pp). The focused Ex1/Ex4 results further show lower Billiards prediction error at all three horizons.

SettingModelBC (%)MPC (%)RMSE@16/32/64 (cm)Selection (%)
Ex1Transformer35.00±2.5035.00±1.800.230/0.750/2.27081.48±1.13
Ex1Pairformer w/o tri.37.00±1.0036.00±1.320.203/0.623/1.82783.45±0.95
Ex1Pairformer39.00±2.7838.00±2.650.190/0.543/1.50785.15±0.88
Ex4Transformer37.50±2.2937.50±1.800.213/0.657/1.88083.50±1.03
Ex4Pairformer w/o tri.39.00±1.8038.83±1.890.190/0.567/1.60084.85±0.97
Ex4Pairformer39.00±2.7838.00±2.650.190/0.543/1.50785.15±0.88

Entries are mean±sample SD over three seeds; BC/MPC use 200 episodes per seed and Billiards selection uses 2,000. Complete RGB-DP configurations contain 74.138M Transformer and 75.643M Pairformer parameters (+2.03%); MPC world models contain 3.594M and 5.098M (+41.86%). The no-triangle ablation retains pair states but removes triangle operations without width compensation; it is not a capacity-matched pair-local control. Ex1 and Ex4 are the parameter- and compute-budget controls, respectively; exact control-specific parameter/FLOP values are not in the outcome records.

W2. MPC cost. A Pairformer world-model step takes 4.907 ms versus 0.870 ms for Transformer (5.64×); training takes 371.6 versus 122.0 GPU-hours (3.05×). Mean full decision-cycle latency is 35.44 s versus 32.27 s (+9.8%; P95: 35.61 s versus 32.38 s). The cycle includes encoding, diffusion sampling, rollout, Value evaluation, and candidate selection, but excludes simulation/rendering and file I/O; it does not demonstrate high-frequency control.

Q1. Protocol and provenance. BC uses 200 demonstrations/task, 50 validation trajectories, three seeds, 300k updates, FP32, batch 64, and EMA selection by validation loss. Reference BC uses 100 episodes/seed (40000–40099); hard-task tests use separate 200-episode banks. MPC uses 200 demonstrations, 50 development trajectories, and 200 evaluation episodes/seed (120000–120199). PegInsertionSide uses the rebuilt content-disjoint split. Focused BC/MPC use 200 episodes/seed; Billiards selection uses 2,000.

Q2. Statistical uncertainty. All seven BC and all seven MPC task means improve (two-sided exact sign test p=.016 for each; task-level, not seed-paired). Ex1 Pairformer−Transformer gains are +4.00 pp BC/+3.00 pp MPC; Ex4 gains are +1.50/+0.50 pp. Unpaired three-seed Welch 95% CIs: Ex1 BC [−2.03,10.03], p=.139; MPC [−2.41,8.41], p=.189; Ex4 BC [−4.36,7.36], p=.512; MPC [−4.91,5.91], p=.802. Billiards RMSE@64 reductions: Ex1 0.763 cm ([0.300,1.226], p=.012); Ex4 0.373 cm ([−0.018,0.765], p=.057). Selection gains: Ex1 +3.67 pp ([1.32,6.02], p=.013); Ex4 +1.65 pp ([−0.53,3.83], p=.103). Welch intervals use three seed aggregates.

Q3. Representation versus planning. BC improves by 6.90 pp (61.10% to 68.00%). With the BC prior and other planner components fixed, Pairformer dynamics improves MPC by 2.85 pp (64.79% to 67.64%). Direct BC to Pairformer MPC is +5.43 pp; direct BC to Transformer MPC is +2.58 pp. Adapted-BC was evaluated only on the development bank. The ensemble penalty (mean predicted improvement minus ensemble SD, coefficient 1) is a ranking heuristic, not calibrated uncertainty.

Q4. VLA generalization. VLA gains concentrate on atomic-seen tasks: 37.88% to 42.11% (+4.23 pp; 18 tasks). Composite-seen changes 10.00%→10.25%; composite-unseen 1.00%→1.25%. These near-floor composite results provide little evidence about compositional improvement. The pair branch receives noisy action-token projections and relative action-time features; visual-language context enters through the retained single stream. Evaluation: three seeds, 50 episodes/task/seed.

Q5. Contact diagnostic. Attention–contact correlation is 0.531 (Transformer) and 0.747 (Pairformer) on one post-hoc-selected held-out layout. In the 300k-trajectory BilliardsRelay evaluation, RMSE@64 decreases 2.960→1.380 cm and selection success increases 79.27%→85.45%; this does not establish cross-layout attention alignment.

Q6. Data-efficiency scope. At 30k trajectories, Transformer/Pairformer RMSE@64 is 2.051/1.627 cm and selection 79.50%/83.10%; at 300k, 2.960/1.380 cm and 79.27%/85.45%. The Transformer result is non-monotonic; these two budgets do not identify why. ManiSkill uses a fixed 200-demonstration budget, so the data-efficiency claim is limited to these settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.