acceptodds
Under review as a conference paper at ICLR 2027

Intervenable Particle Representations for Conditional Image Reconstruction

Abstract

We introduce Particle-U-Net (PUN), a generalization of neural particle automata (NPA). PUN carries particle self-organization from pixel space into the spatial latent space of a frozen non-compressing autoencoder. It injects target-derived features into the particle update via feature-wise linear modulation and reads the resulting particle render into an image with a residual U-Net. NPA learns one rule per target and does not reach high fidelity on natural images. PUN removes both limitations: it handles natural images, generalizes to unseen faces without per-image optimization, and recovers image fidelity that pixel-space fitting does not reach. Beyond reconstruction, we study the explicit particle representation through recovery, coordinate interventions, and component comparisons. Position updates improve recovery from localized damage. Localization after imposed coordinate transformations depends on the spatial FiLM condition during subsequent updates. We also apply directional CLIP guidance to the pixel-domain particle update rule, asking whether the learned dynamics can be steered toward attribute-level edits.

Primary area:
generative models
Keywords:
Neural cellular automata, Particle-based simulation, Conditional image reconstruction, Self-organization, Autoencoders, Generative models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JigcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PUN extends target-specific neural particle automata to shared conditional image reconstruction using a spatial latent, FiLM conditioning, and a residual readout. It studies the resulting particle representation through reconstruction and intervention experiments. The complete model reconstructs 1,000 held-out faces without per-image fitting. Separate interventions bypass the residual U-Net and output conditioning head, showing that particle motion improves recovery and that coordinate responses depend on spatial conditioning.

Critical strengths

The shared-rule formulation is a useful improvement over fitting a separate NPA model to each target. The intervention experiments include controls for particle motion and target conditioning, while state substitution verifies image-specific information. This supports the representation's value as a way to study image formation.

Critical weaknesses

My main concern is the value of particles as an intermediate representation relative to ordinary autoencoder latents or simpler dense alternatives. The interventions demonstrate meaningful dynamics, but do not yet convince me that these provide a sufficiently useful representation for image modeling. The existing dense configuration G reconstructs better than PUN at the same seed (Table 12). The non-compressing design is not itself a weakness; what remains unclear is the intended representational benefit, including whether the states could serve as a latent space for downstream generative modeling.

The evidence is also limited to aligned 64 × 64 CelebA faces. The held-out results validate performance within that setting, while scalability to higher-resolution images and more realistic scenes remains untested. This limits the demonstrated breadth of the representation's usefulness.

Questions for authors

Which existing results demonstrate a representation-level benefit of particles over ordinary autoencoder latents or the dense controls?

Are the particle states intended as reusable features or a latent space for downstream generative modeling, and what current evidence supports that interpretation?

What does the current evidence establish about scalability beyond aligned 64 × 64 faces, and which higher-resolution or realistic settings remain outside the demonstrated scope?

Reviewer_CXynOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Particle-U-Net, a generalization of NPA that offers two advantages: 1) the update rule is no longer per-image (target) but applies to many targets at once; 2) it operates in a non-compression latent space, and it's able to improve the fidelity of reconstructed images. The authors implemented their method on CelebA, showing that PUN not improves image fidelity, but also sustains coordinate (spatial) interventions, compared to other methods.

Critical strengths
  • Extending a generalized, not per-image update rule for particle based methods is a useful direction. The explicit particle representation provides a concrete interface for perturbing states and coordinates.
  • The paper includes informative component controls and intervention experiments. In particular, the mobile-versus-fixed-tail comparison isolates the effect of position updates on localized-damage recovery, while bypassing the residual U-Net and output head. I also appreciate the reporting of dense controls that outperform the full model.
Critical weaknesses
  • My biggest concern is that the contribution of particle dynamics is not clearly separated from direct reconstruction of the conditioning signal. The method gives each particle access to a local sample of the full-resolution target latent, and there's also U-net, and the conditional head also trained to recover the target latent. It's hard to tell whether performance comes from effectiveness of the method, or some information leak through the multiple training components, especially when the latent space is expansive meaning that the whole image can be preserved.
  • Section 4.3 partially addressed the problem, but the recovery task is relatively favorable to their method, and its limited difficulty weakens how much the result establishes. For example, how much learned self-organization is needed to solve this task? Could simple particle redistribution or local reinitialization, followed by state updates, recover similarly? Such a control would retain the ability to refill the damaged region and test whether their learned movement provides something beyond restoring coverage.
  • The headline comparison seems to be with the authors' adapted baselines. Appendix B restricts dense models both in terms of number of parameters and number of iterations, which can be a legitimate comparison under a fixed budget, but requires clear descriptions of the adaptations and evidence that the central baselines train adequately. PUN receives a full-resolution target encoding, whereas DDPM (low-res.) receives a degraded observation; the inputs and objectives for several other baselines are insufficiently specified. It is therefore unclear what's the major advantage of the particle method over the other mode popular methods that the authors compared to, especially when efficiency is a substantial weakness of particle models.
Questions for authors
  • Related to weakness 1), can you give a detailed analysis of the individual contribution of each model component? If the particle network is the major contribution, I'd like to see its performance alone, compared to previous particle-based methods.
  • For my education purpose, can the authors tell me what's the selling point of particle-based methods? Is it that update is local? Or is it recovery? Feel free to talk about it in and out of the context of image reconstruction.
  • Why is there a systematic color shift in target versus PUN in Figure 4? In row 1, 4, and 5.
Reviewer_J4V9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Particle-U-Net (PUN), a conditional version of neural particle automata (NPA). In NPA, particles with continuous positions and internal states are updated by one shared MLP that perceives its neighbours through an SPH operator, the result is rendered by Gaussian splatting, and the rule is trained for a single target image. PUN instead runs 4,096 particles on the 8×64×64 latent grid of a frozen, non-compressing autoencoder. The update MLP is modulated by FiLM parameters computed from the target latent c = Enc(x) at each particle's position, the particle states are splatted back to the latent grid, and the frozen decoder turns them into a render. A residual U-Net refines this render, and an output-side head adds g(c) directly to the output. The particle rule and the readout are trained jointly once on CelebA, without per-image optimization at test time.

On 1,000 held-out aligned 64×64 CelebA faces, PUN reaches a foreground-masked PSNR of 35.03 dB and LPIPS of 0.011, against 20.21 dB for a conditional DDPM given the same latent and 19.93 dB for SiT. All dense baselines are limited to about 1.75M parameters and 30K iterations. The paper then probes the particle representation. Scaling or substituting the particle path shows that the readout uses it. Letting particles move after local deletion or displacement lowers the final error in the damaged region. After a coordinate shift, the content follows the shift only when the FiLM condition is warped in the same way. A last experiment fine-tunes a pixel-domain NPA with a directional CLIP loss and reports which stabilizers let training complete.

Critical strengths
  • The paper removes the main practical restriction of NPA, which is that the rule is trained for one target. A single conditional rule is trained once and applied to unseen faces. The particle render alone, without the U-Net or the output head, reaches 26.20 dB on held-out faces, against 12.13 dB for the original NPA. Sampling the condition at each particle's own position is a simple way to let one rule behave differently across regions of an image, and running the particles in a non-compressing latent keeps the neighbourhood structure the rule depends on. For work on neural cellular and particle automata, this is a step toward rules that generalize across a distribution.

  • The intervention experiments are carefully controlled. Damage recovery compares each damaged branch with a time-matched clean branch that shares its conditioning and random stream, and the mobile, fixed-tail and hold branches isolate the effect of position updates. The coordinate probe separates the role of the conditioning field by keeping, warping or disabling it after the shift. Paired bootstrap intervals are reported throughout. The resulting observations are clear. Position updates drive recovery, with coverage of the damaged region rising from 25.79% to 99.97% within 32 steps. Contiguous deletion is repaired far better than random deletion of the same size.

Critical weaknesses
  • The reconstruction results do not show that the particles contribute to fidelity, and the paper's own ablations suggest they do not. The condition c is a non-compressing encoding of the target, 8×64×64 for a 3×64×64 image from a shallow two-layer encoder, so mapping c back to x is close to an identity task, and the output adds g(c) directly. The frozen decoder alone already gives about 30.3 dB (App. K). A head-only configuration reaches 35.27 dB against 34.92 dB for PUN at the same seed, and the matched arm without particles reaches 36.67 dB against 35.17 dB. With the same U-Net and no head, reading Dec(c) gives 34.42 dB while reading the particle render gives 28.68 dB. Replacing the particle state with that of another image lowers PSNR by only about 3 dB, which is hard to reconcile with the particles carrying the face. Statements such as "recovers image fidelity that pixel-space fitting does not reach" therefore credit the particle framework with fidelity that comes from the readout path. The architecture of g is also not described. A convincing test would use a condition that loses information, such as a low-resolution input or a strongly compressed latent, and remove any path from c to the output that bypasses the particles.

  • The baseline comparison in Table 1 is not informative. The dense baselines are capped at about 1.75M parameters and 30K iterations, and most of them are stochastic generative samplers rather than reconstruction models. Their outputs in Figure 4 are visibly broken, and cDDPM ranges from 17.99 to 23.39 dB across seeds. A 14.82 dB gap over a conditional DDPM that receives the same full-information latent says more about the baseline than about PUN. The relevant baselines are deterministic readouts of c, which the paper runs in the appendix and which outperform PUN. They belong in the main table, and the headline framing should follow from them.

  • The scope is narrow and the cost is high. All results use aligned 64×64 CelebA faces composited on black backgrounds, with an autoencoder trained on 46 synthetic faces, and the component and warm-start comparisons use one seed. Inference takes 635 ms per 64×64 image on a V100, almost all of it in the 192-step rollout, against a single forward pass for the dense readouts that score higher. There is no evidence on how the approach scales with resolution or particle count. Together with the points above, it is unclear what practical or scientific question the reconstruction results answer.

Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.