acceptodds
Withdrawn from ICLR 2027

Atomizer-IO: Beyond Pixels, Patches and Grids

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

77 views

Abstract

Most vision architectures assume that observations lie on a regular grid, an effective abstraction for natural images but a restrictive one for sensing data whose channels, temporal sampling, spatial resolution, and geometry can vary. Generic set-based architectures remove the grid, but also remove useful spatial inductive biases. We introduce Atomizer-IO, an architecture that places observations first and derives structure from their physical relationships. Building on top of an atomic representation of the data, each observation is described by its measurement and acquisition metadata, while local cross-attention maps observations to anchor points that can be arbitrarily placed. We evaluate this design by progressively relaxing the grid assumption, from varying input raster configurations and incomplete channel sets to flexible output density and, ultimately, inputs without a raster grid. Atomizer-IO is competitive with flexible EO-specific architectures on most tasks, while offering post-training control over inference cost and competitive compute–performance trade-offs. The same formulation extends without architectural redesign to unordered 3D point clouds, showing that the atomic interface generalizes beyond regular raster inputs. These results suggest that pixels, patches, and grids do not need to define the interface of a sensing architecture.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Earth observation, vision transformers, set-based learning, multimodal learning, remote sensing, point clouds

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DCBHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Atomizer-IO, a sensing architecture that represents observations as individual measurements with acquisition metadata. Local cross-attention aggregates these observations into spatially anchored latent representations, which interact through a Transformer and are decoded at requested output locations. This design supports variable input channels, irregular geometries, and adjustable output density. The paper evaluates the architecture through separately trained models on eight Earth-observation benchmarks and two LiDAR datasets, covering classification, regression, and segmentation.

Critical strengths
  1. The paper is well motivated and proposes a reasonable solution that aligns with the identified challenges.

  2. The paper is clearly written and easy to follow.

Critical weaknesses
  1. The methodological advancements of this approach over existing frameworks have not yet been fully substantiated. The architecture combines atomic representations, latent cross-attention, query-based decoding, and local geometric aggregation. While this combination is meaningful, the paper does not explicitly clarify what substantive capabilities or insights it adds beyond Atomizer, Perceiver IO, and existing geometry-aware aggregation methods. It appears to be a blend of many concepts, but there are no detailed experiments to support that this combination demonstrates meaningful synergy between the components. This limits the strength of its core contribution.

  2. The discussion regarding the relationship with graph-based methods is insufficient. Designs such as local neighborhoods and anchor aggregation share similarities with information aggregation on graphs; the paper needs to clarify the connections and substantive differences.

  3. The point cloud baselines are insufficient. While KPConv was mentioned as an earlier method, the paper should include newer, strong baselines such as PointNeXt and Point Transformer V3 for comparison under fair conditions.

Questions for authors
  1. What is the substantive advance over Atomizer and Perceiver IO beyond incorporating local geometric aggregation?
  2. What evidence shows that the unified interface offers practical advantages over task-specific architectures when accounting for both accuracy and computational cost?
3 comments

Official Response to Reviewer DCBH

We thank Reviewer DCBH for the feedback. After considering the reviews, we have decided to withdraw the submission from ICLR 2027. We nevertheless provide this response to clarify the contribution and to document how the feedback will inform a future revision. We address the main concerns below and point to the relevant experiments and analyses already included in the paper and Appendix.

W1 & Q1 — Substantive advance over Atomizer / Perceiver-IO and component synergy

What is the substantive advance over Atomizer and Perceiver IO beyond incorporating local geometric aggregation? ... there are no detailed experiments to support that this combination demonstrates meaningful synergy between the components.

First, the advance over Atomizer (2025) and Perceiver-IO goes beyond adding local geometric aggregation. Atomizer introduced the atomic representation, but was designed only as an encoder for scene-level classification and was evaluated on a synthetic benchmark. Atomizer-IO changes how this representation is structured, scaled, and decoded:

Atomizer (2025)Atomizer-IO
ArchitectureEncoder onlyEncoder–processor–decoder
Spatial positionAbsoluteRelative
Latent allocationFixedScales with input observations
Input aggregationGlobal cross-attentionLocal spatial cross-attention
Input–latent costO(LN)O(LN)O(N)O(N) without subsampling
OutputScene-level predictionArbitrary spatial queries
Tasks evaluatedClassificationClassification, regression, 2D/3D segmentation
Temporal processingNot evaluatedJoint or explicit temporal aggregation
Inference controlFixedOutput density adjustable after training

In particular, Atomizer's global cross-attention requires every latent to attend to every input atom. On a single 5122×15512^2 \times 15 Sen1Floods11 input with L=2000L=2000 latents, this would require approximately 7.86×1097.86\times10^9 input–latent interactions. Atomizer-IO instead assigns each atom to a local spatial latent, reducing this to 3.93×1063.93\times10^6 interactions (99.95% reduction; Appendix B.7, Table 7). The number of spatial latents also scales with the number of observations, so additional input information can be allocated additional latent capacity without changing the model parameters.

Together with relative spatial encoding and a spatial query decoder, these changes extend the atomic representation from scene classification to regression, multitemporal prediction, and dense 2D and 3D segmentation. The query-based output interface also allows the same trained checkpoint to vary its output density after training, providing a direct accuracy–computation trade-off (Section 4.3).

Comparison with Perceiver-IO

Both architectures use an encoder–processor–decoder organization and support query-based outputs. The distinction is how information is organized and accessed:

Perceiver-IOAtomizer-IO
Encoder aggregationGlobal input–latent attentionLocal attention around spatial anchors
Latent representationNo assigned spatial positionsSpatial anchors with relative geometry (RoPE)
Latent allocationFixed latent countScales with input observations
Decoder latent accessGlobal latent arrayFixed number of nearby spatial latents

For raster inputs, tokenization differs too: Perceiver-IO groups channels into pixel-level tokens, whereas Atomizer-IO represents each measurement independently. Although training protocols and metadata encodings are matched, this comparison evaluates the complete architectures rather than isolating spatial anchoring alone.

Both models can also condition predictions directly on observations at the queried location. We remove this pathway to test their dependence on local evidence when decoding from the latent representation:

ConfigurationPASTISBurnScarsSen1Floods11DALES
Perceiver-IO17.983.492.248.7
Perceiver-IO w/o local observations17.970.675.8–
Atomizer-IO44.188.893.266.7
Atomizer-IO w/o local observations43.388.289.3–

Removing local observations reduces Perceiver-IO performance by 12.8 points on BurnScars and 16.4 on Sen1Floods11, compared with 0.6 and 3.9 for Atomizer-IO. Without this pathway, Atomizer-IO retains 88.2 and 89.3 mIoU, respectively, versus 70.6 and 75.8 for Perceiver-IO. These results suggest that Atomizer-IO better preserves information needed for prediction in its latent representation on these tasks. On PASTIS, removing local observations changes neither model substantially; the performance gap remains, but this experiment alone does not identify its cause.

DALES provides a complementary comparison: both models receive one token per LiDAR return, removing the raster tokenization difference. Atomizer-IO reaches 66.7 mIoU versus 48.7 for Perceiver-IO. This supports the effectiveness of its spatial architecture on unordered inputs, although aggregation, latent allocation, and decoding also differ.

Second, regarding the contribution of the individual components, Appendix B already provides controlled ablations of the main sources of structure introduced by Atomizer-IO. These are independent ablations rather than a sequential addition of components:

  • Local spatial encoding: reduces the input–latent interaction count from 7.86×1097.86\times10^9 to 3.93×1063.93\times10^6 without subsampling.
  • Spatial RoPE (Appendix B.2): removing it reduces Cashew from 72.01 to 62.65 mIoU and MNIST from 99.37 to 29.48% accuracy.
  • Target-local observation conditioning (Appendix B.3): removing it reduces Sen1Floods11 from 93.17 to 89.26 mIoU, with smaller effects on tasks relying less strongly on target-local measurements.
  • Temporal aggregation (Appendix B.1): improves PASTIS by +0.47, +4.85, and +8.58 mIoU at 3, 6, and 12 timesteps, respectively.

These results show that local aggregation, relative spatial structure, target-local information, and temporal organization address different aspects of the problem rather than acting as redundant additions. We will make this analysis more visible in a future revision.

ConfigurationMNISTCashewPASTIS (T=6)BurnScarsSen1Floods11
Atomizer-IO w/o Spatial RoPE29.4862.6540.6586.1692.42
Atomizer-IO w/o Local Obs.–70.5843.3088.2089.26
Atomizer-IO w/o Temporal Agg.––39.25––
Atomizer-IO99.3772.0144.1088.8193.17

W2 — Relationship and substantive differences with graph-based methods

Designs such as local neighborhoods and anchor aggregation share similarities with information aggregation on graphs; the paper needs to clarify the connections and substantive differences.

We will expand this discussion in a future revision. Mathematically, the Spatial Sampler can be viewed as constructing a bipartite geometric graph: in the encoder, atoms connect to their nearest spatial anchors; in the decoder, output queries connect to their nearest spatial anchors.

There are three main differences from standard GNNs and 3D superpoint graphs:

  1. No atom-to-atom edge construction. Standard graph formulations commonly construct local edges between input nodes. For the approximately 3.93×1063.93\times10^6 atoms of a Sen1Floods11 input, this would create a very large input graph. Moreover, multiple spectral measurements share exactly the same spatial coordinate. Atomizer-IO instead maps atoms directly to a much smaller set of spatial anchors.

  2. The raster partition can be precomputed. For fixed geometries, the Voronoi assignment depends only on coordinates and can therefore be cached. In contrast, superpoint methods such as SPG/SPT use data-dependent geometric grouping.

  3. Global communication occurs in the latent processor. Rather than propagating information through repeated local graph hops, Atomizer-IO applies global self-attention over the spatial latents, with their geometry represented through 2D RoPE.

W3 — Point-cloud baselines and 3D performance analysis

The point cloud baselines are insufficient. While KPConv was mentioned as an earlier method, the paper should include newer, strong baselines such as PointNeXt and Point Transformer V3 for comparison under fair conditions.

We agree that the comparison should include more recent point-cloud architectures. Table 4 already includes Superpoint Transformer (ICCV 2023), and in response to the reviewers we additionally evaluated Point Transformer V3 on FRACTAL.

We provide PTv3 with both modalities by associating VHR RGB values with the LiDAR points as fixed per-point features. PTv3 reaches 80.12% mIoU on the test set, compared with 78.4% for Atomizer-IO.

This result is consistent with the intended role of the 3D experiments. We do not claim state-of-the-art point-cloud segmentation. FRACTAL tests whether the same architecture can jointly process image and unordered LiDAR measurements without requiring them to first share a per-point feature representation. DALES goes one step further and tests the same Atomizer-IO spatial interface on a pure point cloud, without introducing a 3D-specific backbone.

The small FRACTAL gap to PTv3 shows that a specialized modern point-cloud architecture can perform slightly better when given an explicit RGB-to-point correspondence, while Atomizer-IO remains close using its more general independent-measurement interface. We will broaden this comparison further in a future revision.

Q2 — Practical advantages over task-specific architectures

What evidence shows that the unified interface offers practical advantages over task-specific architectures when accounting for both accuracy and computational cost?

Our goal is not to claim that a unified architecture should always outperform specialized models. Rather, we ask whether supporting heterogeneous inputs, irregular geometries, and flexible outputs comes at a substantial cost in accuracy or computation.

To isolate the effect of the architecture itself, all models in our main comparisons are trained from scratch under matched training conditions. We deliberately avoid pretrained weights because pretraining would make it difficult to separate gains coming from the architecture from gains coming from the amount or nature of pretraining data.

Under this controlled setting:

  1. Competitive accuracy under matched training conditions (Table 2). Atomizer-IO ranks first on four and second on two of the six dense/spatiotemporal EO benchmarks, while using the same architecture across classification, regression, 2D segmentation, multitemporal prediction, and irregular inputs. This suggests that the unified interface does not impose a large accuracy penalty on standard raster tasks.

  2. Graceful adaptation to changing sensor configurations (Table 3). A single trained checkpoint can be evaluated with different subsets of sensors or spectral bands simply by removing the corresponding atoms. No padding, missing-channel mask, or architectural modification is required.

  3. Post-training control of inference cost (Section 4.3, Figure 5, Appendix C). Because predictions are produced through spatial queries, the number and location of output queries can be changed after training. The same checkpoint can therefore trade output density for decoder computation using dense, zone-probe, or quadtree querying without retraining.

The results also show where this flexibility has a cost. On DALES, specialized 3D architectures such as KPConv and SPT outperform Atomizer-IO, and on FRACTAL the newly evaluated PTv3 reaches 80.12% mIoU compared with 78.4% for Atomizer-IO. Atomizer-IO deliberately keeps the same general spatial interface used for 2D sensing rather than introducing a task-specific 3D backbone.

We view this as an important part of the evaluation: the purpose of the paper is to characterize when a unified sensing interface remains competitive, where it provides additional flexibility, and where stronger task-specific inductive biases are still beneficial.

Reviewer_ho2tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a neural architecture to process unstructured data, embedded in an n-dimensional space. The direct application is earth-observation data, where the observations are embedded in 2/3-dimensional space and may contain spectral responses or distances from the sensor. Often, some spectral bands are missing, the capture resolution differs between datasets, or the samples are observed in an unstructured way. Therefore, the paper treats each observation (measurement + context) as a unit (atom), rather than assuming a fixed context (e.g., grid sampling on a fixed set of bands), which I understand to be the main contribution of Atomizer [de Turckheim et al. 2025], not the current work.

Atomizer-IO efficiently processes these atoms by introducing an encoder and decoder and processing only a small set of latents. The encoder/decoder idea is typical in learning (e.g., Perceiver-IO, DiT), but the paper claims contribution on using an encoder with cross-attention between the input observations and a small set of spatially anchored latents (which to me is reminiscent of structured latents, e.g., TRELLIS https://microsoft.github.io/TRELLIS/ and unstructured latents, e.g., 3DShape2VecSet).

The approach is validated in several settings, starting from grid-based data and increasingly reducing structure. In the grid-based setting, the paper outperforms other work when spatial structure is required. In a setting where input channels are removed, the paper outperforms all other approaches on average. The authors show that the output can be queried at arbitrary locations, yielding a speed-up combined with a probing technique. In a setting where the grid is removed (general point clouds), the paper is better than Perceiver-IO, but significantly worse than competing point-cloud methods (-15 mIoU points than KPConv in DALES, only RanLA-Net is compared for FRACTAL).

Critical strengths
  • The approach makes sense to me and I appreciate that a simple mechanism can work for increasingly unstructured settings as well as structured settings.
  • The application setting is important and the problem of working with different bands is valuable. The paper shows this relevance well, through its exposition and experiments.
Critical weaknesses

Many of the contributions are already present in prior work. For example, the atomic representation is presented in Atomizer [de Turckheim et al. 2025], the encoder/processor/decoder structure is present in Perceiver-IO [Jaegle et al. 2021a], the spatial aggregation and cross-attention is common in point cloud learning (e.g., PointTransformer [Zhao et al. 2021]) and shape representation (e.g., VecSet, TRELLIS). The paper claims "a physical grounded atomic representation" as one of the contributions, but how is this different from Atomizer? My understanding is that the combination of all these components gives the performance boost in the experiments, but it provides no fundamentally new technical insight.

The results on point cloud understanding (table 4) are rather concerning to me. The paper claims that "Atomizer-IO is competitive on both datasets", but 1) Atomizer-IO is 15 mIoU points below KPConv [2019] for DALES, 2) Only Rand-LA-Net [2020] is compared for FRACTAL 3) the most recent compared method is from 2023, while more recent point-cloud based approaches are available (e.g., KPConvX, PointTransformerv3). The positioning of the paper compared to point-cloud methods is that "[...] these approaches are not designed to jointly accommodate multi-spectral, multi-temporal, and multi-resolution data," (l. 141); however, it seems straightforward to apply point-cloud methods on multi-spectral and multi-resolution data. For example, most point-cloud methods can take an arbitrary number of input channels. If robustness to dropped channels is important, is it non-trivial to apply the existing Atomizer concept within these settings?

I am not sure of this point, so I'd be interested in the authors take on this: in figure 6, left, ViT has much better cost-quality comparison. What is the behavior of scaling up ViT?

The following are not critical weaknesses and concern writing, but addressing them would make the paper easier to read:

  • The exposition on the spatial sampler appears to be needlessly convoluted, as it is a nearest neighbor lookup on point samples. It does not appear to require multiple paragraphs and notations (V_l = A_enc = nearest neighbor lookup).
  • The paper refers to the Appendix, where a simple notation in the paper itself could work (e.g., 226, 259)
  • l. 249,250: global latent "in spirit to register tokens". Clarify directly why a global latent is used, rather than resorting to a reference.
  • Missing reference for ViT (l. 322)
  • Table 1, 2, and 3 display the methods/datasets in different orders. Please match them for easy comparison.
Questions for authors
  • How do the authors see the relation to Atomizer/Perceiver IO?
  • What ViT is used? The reference is missing in l. 322 and it is unclear what settings are compared to.
  • Why does row 1 in tab. 3 not match with columns 'BurnScars' and 'Biomassters' in tab. 2?
  • Sensitivity to sampling? What if you train on uniform sampling and test on heterogeneous sampling? My expectation is that the decoder may struggle, because the latents likely encode somewhat similar patches
3 comments

Official Response to Reviewer ho2t

We thank Reviewer ho2t for the feedback. After considering the reviews, we have decided to withdraw the submission from ICLR 2027. We nevertheless provide this response to clarify the contribution and to document how the feedback will inform a future revision. We agree that several individual components have precedents. Our contribution is not a new attention primitive, but an interface that separates what is observed from how space is structured: measurements are represented independently, organized into a scalable spatial latent representation, and decoded at arbitrary spatial locations.

W1 & Q1 — Relation to Atomizer, Perceiver-IO, and structured latent representations

Many of the contributions are already present in prior work. [...] How do the authors see the relation to Atomizer/Perceiver IO?

We agree that the individual ingredients have precedents, and in particular that the atomic representation itself was introduced in Atomizer (2025). We will revise the contribution statement to make this more explicit in a future revision. Atomizer-IO instead extends this representation into a scalable spatial input/output architecture.

Beyond Atomizer

Atomizer was evaluated only as an encoder for single-timestep scene classification on a synthetic benchmark. Atomizer-IO changes how the atomic representation is structured, scaled, and decoded:

Atomizer (2025)Atomizer-IO
ArchitectureEncoder onlyEncoder–processor–decoder
Spatial positionAbsoluteRelative
Latent allocationFixedScales with input observations
Input aggregationGlobal cross-attentionLocal spatial cross-attention
Input–latent costO(LN)O(LN)O(N)O(N) without subsampling
OutputScene-levelArbitrary spatial queries
Tasks evaluatedClassificationClassification, regression, single-/multitemporal 2D segmentation, 3D segmentation

Local aggregation reduces input–latent attention interactions by more than 99% on dense inputs (Appendix B.7). Together with adaptive spatial latents and query-based decoding, this extends Atomizer from scene classification to dense spatial prediction.

Beyond Perceiver-IO

Both architectures use an encoder–processor–decoder organization and support query-based outputs. The distinction is how information is organized and accessed:

Perceiver-IOAtomizer-IO
Encoder aggregationGlobal input–latent attentionLocal attention around spatial anchors
Latent representationNo assigned spatial positionsSpatial anchors with relative geometry (RoPE)
Latent allocationFixed latent countScales with input observations
Decoder latent accessGlobal latent arrayFixed number of nearby spatial latents

For raster inputs, tokenization differs too: Perceiver-IO groups channels into pixel-level tokens, whereas Atomizer-IO represents each measurement independently. Although training protocols and metadata encodings are matched, this comparison evaluates the complete architectures rather than isolating spatial anchoring alone.

Both models can also condition predictions directly on observations at the queried location. We remove this pathway to test their dependence on local evidence when decoding from the latent representation:

ConfigurationPASTISBurnScarsSen1Floods11DALES
Perceiver-IO17.983.492.248.7
Perceiver-IO w/o local observations17.970.675.8–
Atomizer-IO44.188.893.266.7
Atomizer-IO w/o local observations43.388.289.3–

Removing local observations reduces Perceiver-IO performance by 12.8 points on BurnScars and 16.4 on Sen1Floods11, compared with 0.6 and 3.9 for Atomizer-IO. Without this pathway, Atomizer-IO retains 88.2 and 89.3 mIoU, respectively, versus 70.6 and 75.8 for Perceiver-IO. These results suggest that Atomizer-IO better preserves information needed for prediction in its latent representation on these tasks. On PASTIS, removing local observations changes neither model substantially; the performance gap remains, but this experiment alone does not identify its cause.

DALES provides a complementary comparison: both models receive one token per LiDAR return, removing the raster tokenization difference. Atomizer-IO reaches 66.7 mIoU versus 48.7 for Perceiver-IO. This supports the effectiveness of its spatial architecture on unordered inputs, although aggregation, latent allocation, and decoding also differ.

Relation to TRELLIS, 3DShape2VecSet, and point-cloud methods

We thank the reviewer for raising this point, which gives us the opportunity to clarify a potentially confusing distinction.

We agree that spatial and set-based latent representations are not new by themselves. TRELLIS uses a different construction: it places latents on a sparse 3D voxel grid and builds their features from pretrained DINOv2 features extracted from multiple rendered views of the object. The latent structure is therefore tied to a predefined 3D discretization and object occupancy. 3DShape2VecSet similarly learns latent sets for representing 3D shapes. These representations are designed for 3D shape encoding and generation rather than for organizing heterogeneous physical measurements.

Point-cloud architectures naturally handle a variable number of irregularly located points, but typically assume the same fixed feature schema for every point. Consider FRACTAL: the VHR image and LiDAR can be acquired several years apart, so there is no natural one-to-one correspondence between an image measurement and a LiDAR point. Similarly, in multitemporal EO, several measurements acquired at different times can refer to the same spatial location. Representing these observations as one feature vector per point therefore requires deciding in advance which measurements should be grouped together.

Atomizer-IO instead represents each measurement independently. Its sensing properties are encoded through acquisition metadata, while spatial position is used separately to define locality. Measurements from different sensors, resolutions, and timesteps can therefore coexist without requiring one-to-one correspondence between them.

The contribution is therefore not the use of spatial latents itself, but using them to turn the atomic representation into a scalable spatial architecture that can process heterogeneous sensing inputs and produce spatial outputs.

W2 — Point-cloud baselines and 3D positioning

The results on point cloud understanding are rather concerning to me. [...] More recent point-cloud based approaches are available.

We agree that the wording “competitive on both datasets” is too broad and will revise it in a future revision. The 3D experiments are intended as two complementary stress tests of the same architecture. FRACTAL remains close to the 2D setting, but adds unordered LiDAR measurements to VHR imagery and therefore tests multimodal fusion with irregular 3D observations. DALES goes one step further and removes the 2D image structure entirely, testing Atomizer-IO on a pure point-cloud input without introducing a 3D-specific backbone.

In response to the reviewer, we additionally evaluated Point Transformer V3 on FRACTAL. To provide both modalities to PTv3, VHR RGB values are associated with the LiDAR points and included as fixed per-point features. PTv3 reaches 80.12 mIoU on the test set, compared with 78.4 for Atomizer-IO.

This is consistent with our intended positioning: a specialized modern 3D architecture can slightly outperform Atomizer-IO when image information is first mapped into its per-point feature representation, whereas Atomizer-IO processes the VHR and LiDAR measurements independently through the same general sensing interface. We therefore do not claim state-of-the-art 3D performance.

W3 — Cost–quality of ViT and scaling

In figure 6, left, ViT has much better cost-quality comparison. What is the behavior of scaling up ViT?

The baseline is ViT-Small. Its lower cost at the operating point shown in Figure 6 is expected, since ViT can exploit the complete regular raster through patch-based processing. Figure 6 does not characterize how ViT scales as its spatial granularity is increased, and we did not complete a dedicated scaling study before withdrawing. We will make this limitation explicit and study this comparison more systematically in a future revision.

We note that ViT's efficiency also comes with a more restrictive input interface: when SWIR bands are removed in Table 3, ViT drops to 75.1 mIoU compared with 87.8 for Atomizer-IO, despite both being trained with the same band-dropout augmentation.

Q4 — Sensitivity to sampling shift

What if you train on uniform sampling and test on heterogeneous sampling? My expectation is that the decoder may struggle, because the latents likely encode somewhat similar patches.

Table 6 directly tests this type of shift on the input geometry. Atomizer-IO is trained only on complete, regularly sampled MNIST images and evaluated with the same checkpoint on increasingly sparse and irregular inputs. With only 25% of background observations retained, it still reaches 94.9% accuracy, compared with 72.2% for the parameter-matched ViT.

Because MNIST is a classification task, this experiment does not directly test the spatial decoder. Appendix C.2 provides complementary evidence on dense prediction: the same Sen1Floods11 and BurnScars checkpoints are evaluated under substantial changes in latent density and decoder neighborhoods without retraining. These changes are spatially uniform, however. We therefore consider robustness to arbitrary heterogeneous sampling patterns in dense prediction an interesting open question to study in a future revision.

Q2 — ViT configuration

What ViT is used? The reference is missing in l. 322 and it is unclear what settings are compared to.

We use ViT-Small for all 2D experiments. For dense prediction it is paired with a UPerNet decoder, and for multitemporal tasks with L-TAE, following the PANGAEA protocol. All architectures are trained from scratch under the same task-specific training protocol and are selected to operate in a similar parameter regime; full parameter counts and training details are reported in Appendix D.2. We will make the ViT configuration and reference explicit in the main text in a future revision.

Q3 — Difference between Tables 2 and 3

Why does row 1 in tab. 3 not match with columns “BurnScars” and “BioMassters” in tab. 2?

Table 3 uses checkpoints trained with band-dropout augmentation, whereas Table 2 reports the standard single-task checkpoints. This explains the small difference even when all bands are available at test time.

Importantly, the same band-dropout augmentation is used for all architectures, including the fixed-channel baselines. This gives every model explicit training exposure to missing-band configurations and avoids disadvantaging baselines by testing them under an input condition they were not trained for. We will clarify this distinction in the table caption and main text in a future revision.

Reviewer_ZqdwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a flexible vision architecture to handle heterogeneous and irregular sensing data. The method represents each individual measurement, along with its acquisition metadata, as an atomic token. A local attention mechanism aggregates these tokens into spatially anchored latent vectors, which are then processed by a Transformer and decoded with spatial queries.

The main contribution is the unified encoder-processer-decoder architecture that supports various input and output schema for different source of data and tasks without architectural modifications.

The authors evaluate their method mainly on Earth observation benchmarks, achieving competitive results compared with baselines. The method is also tested when input is broken or irregular, to further show its flexibility.

Critical strengths
  1. The architectural design is reasonable. By providing a unified representation for various I/O schemas, the proposed architecture can support multiple vision tasks without architectural modifications.

  2. The results are encouraging. The same architecture achieves competitive performance on several Earth objservation benchmarks, indicating its potential as a general framework for diverse sensing tasks.

Critical weaknesses
  1. The proposed architecture largely combines existing techniques, including atometic representation from Atomizer, and the encoder-processor-decoder design from Perceiver IO. The irregular set to structured latent technique also has widely been explored in pointcloud networks.

  2. Validation seems not sufficient enough. While separate models are trained for different datasets and tasks, it remains unclear whether a single model can handle heterogeneous inputs. Several complex design choices also lack comparisons with simpler alternatives and are thus less convincing.

Questions for authors
  1. Since aggregating irregular set to structured latents has been widely explored in spatial networks, what is the unique advantage of the proposed approch over existing techniques?

  2. Can a single model be trained and tested on various sources of data for multiple tasks?

  3. The Voronoi partitioning makes the system more complex. What advantages does it offer over simpler grid-based partitioning, which can be conveniently processed with regular CNNs or ViTs?

3 comments

We thank Reviewer Zqdw for the feedback. After considering the reviews, we have decided to withdraw the submission from ICLR 2027. We nevertheless provide this response to clarify the contribution and to document how the feedback will inform a future revision. We agree that several individual components have precedents. Our contribution is not a new attention primitive, but an interface that separates what is observed from how space is structured: measurements are represented independently, organized into a scalable spatial latent representation, and decoded at arbitrary spatial locations.

W1 & Q1 — Unique advantage over Atomizer, Perceiver-IO, and point-cloud networks

Since aggregating irregular set to structured latents has been widely explored in spatial networks, what is the unique advantage of the proposed approach over existing techniques?

1. Beyond Atomizer: from a classification encoder to a scalable spatial architecture

Atomizer (2025) introduced the atomic representation, but was evaluated only as an encoder for single-timestep scene classification on a synthetic benchmark. Atomizer-IO changes how this representation is structured, scaled, and decoded:

Atomizer (2025)Atomizer-IO
ArchitectureEncoder onlyEncoder–processor–decoder
Spatial positionAbsoluteRelative
Latent allocationFixedScales with input observations
Input aggregationGlobal cross-attentionLocal spatial cross-attention
Input–latent costO(LN)O(LN)O(N)O(N) without subsampling
OutputScene-level predictionArbitrary spatial queries
Tasks evaluatedClassificationClassification, regression, single-/multitemporal 2D segmentation, 3D segmentation

In particular, replacing global input–latent attention with local aggregation reduces attention interactions by more than 99% on dense inputs (Appendix B.7), making the atomic representation practical for spatial prediction. Together with adaptive spatial latents, relative geometry, and query-based decoding, this extends Atomizer from a scene-classification encoder to a general spatial input/output architecture.

2. Beyond Perceiver-IO: a spatially organized atomic architecture

Both architectures use an encoder–processor–decoder organization and support query-based outputs. The distinction is how information is organized and accessed:

Perceiver-IOAtomizer-IO
Encoder aggregationGlobal input–latent attentionLocal attention around spatial anchors
Latent representationNo assigned spatial positionsSpatial anchors with relative geometry (RoPE)
Latent allocationFixed latent countScales with input observations
Decoder latent accessGlobal latent arrayFixed number of nearby spatial latents

For raster inputs, tokenization differs too: Perceiver-IO groups channels into pixel-level tokens, whereas Atomizer-IO represents each measurement independently. Although training protocols and metadata encodings are matched, this comparison evaluates the complete architectures rather than isolating spatial anchoring alone.

Both models can also condition predictions directly on observations at the queried location. We remove this pathway to test their dependence on local evidence when decoding from the latent representation:

ConfigurationPASTISBurnScarsSen1Floods11DALES
Perceiver-IO17.983.492.248.7
Perceiver-IO w/o local observations17.970.675.8–
Atomizer-IO44.188.893.266.7
Atomizer-IO w/o local observations43.388.289.3–

Removing local observations reduces Perceiver-IO performance by 12.8 points on BurnScars and 16.4 on Sen1Floods11, compared with 0.6 and 3.9 for Atomizer-IO. Without this pathway, Atomizer-IO retains 88.2 and 89.3 mIoU, respectively, versus 70.6 and 75.8 for Perceiver-IO. These results suggest that Atomizer-IO better preserves information needed for prediction in its latent representation on these tasks. On PASTIS, removing local observations changes neither model substantially; the performance gap remains, but this experiment alone does not identify its cause.

DALES provides a complementary comparison: both models receive one token per LiDAR return, removing the raster tokenization difference. Atomizer-IO reaches 66.7 mIoU versus 48.7 for Perceiver-IO. This supports the effectiveness of its spatial architecture on unordered inputs, although aggregation, latent allocation, and decoding also differ.

3. Difference from point-cloud and spatial set-to-latent networks

We agree that mapping irregular sets to spatially structured representations is well established, and we do not claim this operation itself as novel. The difference is in what is represented and how spatial structure is defined.

Point-cloud architectures typically represent each 3D point together with a fixed-dimensional feature vector, with the same feature schema shared by all points. For example, every point may carry RGB, intensity, or other attributes with predefined semantics.

This assumption does not hold naturally in our setting. Remote-sensing observations can come from different sensors with different measurement types and metadata. Sen1Floods11 combines Sentinel-1 and Sentinel-2, while FRACTAL combines VHR imagery and LiDAR whose acquisitions can be separated by several years. More broadly, Table 1 spans very different sensing modalities and channel configurations. Representing all of these observations as standard point features would therefore require defining in advance a common feature layout across sensors and modalities.

Atomizer-IO instead represents each measurement independently. Its identity is described by acquisition metadata, while spatial position is used separately to define locality. Different sensors, bands, and timesteps can therefore coexist in the same representation without requiring a shared predefined feature schema or exact one-to-one correspondence between observations.

The advance is therefore not set-to-latent aggregation itself, but combining independent measurement tokens with a scalable spatial representation and query-based decoding. This lets the same architecture process regular images, multitemporal data, different sensor configurations, and unordered point clouds.

W2 & Q2 — Can a single model handle heterogeneous sources of data?

While separate models are trained for different datasets and tasks, it remains unclear whether a single model can handle heterogeneous inputs... Can a single model be trained and tested on various sources of data for multiple tasks?

Table 2 uses a separate checkpoint for each benchmark; we do not claim a single jointly trained multi-task model, and we will revise the wording of our first contribution accordingly in a future revision. Our claim is that the same architecture, without input-specific modules, can process heterogeneous sensing inputs without changing its input interface. We demonstrate this within single checkpoints in three settings:

  • RGB imagery and unordered LiDAR jointly (FRACTAL, Section 4.4). Each sample combines VHR RGB imagery with 40,000 unordered LiDAR points. The two acquisitions can be separated by several years, making exact point-to-image correspondence difficult. Both modalities are nevertheless converted to atoms, merged into the same input set, and processed by the same encoder weights, reaching 78.4 mIoU.

  • Different sensor subsets with one checkpoint (Table 3). The same trained model is evaluated with S1 only, S2 only, RGB only, and joint sensor configurations. Unavailable measurements simply produce no atoms: no zero-padding, missing-channel mask, or change in atomic feature dimensionality is required. All models in this experiment, including the fixed-channel baselines, were trained with the same band-dropout augmentation, so the comparison reflects how each architecture handles missing measurements rather than whether it saw them during training.

  • Unseen input geometries with one checkpoint (Table 6). On MNIST, a checkpoint trained only on complete images is evaluated with progressively sparser, irregular inputs. With 25% of background pixels retained, Atomizer-IO reaches 94.9% accuracy, compared with 72.2% for a parameter-matched ViT given exactly the same observations.

Beyond these settings, the same architecture is trained without input-specific modifications across different channel counts, spatial dimensions, temporal lengths, and geometries (Tables 1–2). Joint training across datasets and tasks is a natural next step, and is precisely the setting that motivates our design. We see this as an important direction for a future revision toward multi-sensor EO foundation models.

W2 (cont.) — Comparisons with simpler alternatives

Several complex design choices also lack comparisons with simpler alternatives and are thus less convincing.

Each main design choice is compared with a simpler alternative in the paper and Appendix:

  • Spatial atomic architecture vs. a global-latent baseline: Perceiver-IO under matched training protocols and metadata encodings (Table 2 and the comparison above); raster tokenization also differs.
  • Relative spatial structure between latents vs. none: spatial RoPE ablation (Appendix B.2).
  • Target-local conditioning vs. latent-only decoding: local observation ablation (Appendix B.3).
  • Explicit temporal aggregation vs. joint encoding of all timesteps: gains of +0.5, +4.9, and +8.6 mIoU on PASTIS at 3, 6, and 12 timesteps.
  • Latent density, encoder attention budget, and decoder neighborhood size: compute–accuracy sensitivity analysis (Appendix C).

We will make these comparisons more visible in the main text in a future revision.

W3 & Q3 — Voronoi partitioning vs. grid-based partitioning

The Voronoi partitioning makes the system more complex. What advantages does it offer over simpler grid-based partitioning, which can be conveniently processed with regular CNNs or ViTs?

  1. A regular grid is a special case of Voronoi partitioning. When anchors are arranged on a regular grid, nearest-anchor assignment recovers regular spatial cells. When the input geometry is fixed, the assignments depend only on coordinates and can be precomputed and cached, making the overhead of the Voronoi partition negligible (Appendix B.7).

  2. The same rule works when observations are no longer on a grid. Point clouds, sparsified images, or variable measurement sets are assigned to their nearest spatial anchors without changing the architecture or forcing the observations onto a dense tensor.

  3. Grid cells processed by CNNs or ViTs require a fixed tensor layout. A CNN or ViT operating on grid cells expects the same channels at every location, so missing measurements must be zero-padded. In Atomizer-IO, a cell simply contains whichever atoms are present. Table 3 shows the cost of padding: on Sen1Floods11 with SWIR bands removed, ResNet50 drops to 49.0 mIoU and ViT to 75.1, while Atomizer-IO retains 87.8, with all models trained under the same band-dropout augmentation.

The purpose of the Voronoi sampler is therefore not to improve on grid partitioning when a regular grid is available, but to provide one spatial interface that recovers the grid case while also handling irregular geometries and variable measurement sets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.