acceptodds
Under review as a conference paper at ICLR 2027

TOKAFORMER: ROUTING CO-LOCATED MODALITIES AS TYPED TOKENS BEFORE COMPRESSION

Abstract

Multimodal models often fuse the inputs that share a position by embedding each with one shared encoder and adding the results. This is cheap, but when track identity enters only through additive offsets it erases the pairing between each value and the input it came from, and tasks that must report what one particular input said cannot recover it. We asked when this loss matters and how to avoid it at reasonable cost. For such a shared additive encoding we prove that every reassignment of values among inputs yields the same fused vector, so no downstream model can retrieve a named input's value better than a histogram bound we compute exactly. A code that gives each input its own subspace escapes the theorem, which locates the problem in the encoder rather than in addition. Tokaformer, the architecture we study, keeps each position's inputs as separate typed tokens through a linear-recurrent routing stage and only then discards the extra tokens. On position-, content-, and track-addressed synthetic tasks, factored input succeeded where summed input failed, and the failure was not explained by sequence length or shared tuning. On aggregation tasks such as counting, neither fusion held a robust advantage, which suggests pooling only the inputs an aggregate can absorb; this hybrid reached similar observed composite accuracy to full factoring (about 0.996) with 35% fewer routing tokens. On WeatherBench 2 and BigEarthNet the ranking of the two fusions depended on the input encoder: with one encoder shared across channels factored fusion won clearly, while with a separate encoder per channel, which binds identity to value before the sum, summed fusion was as good or better, consistent with the proposed mechanism. We summarize the results as bind before you compress: keep inputs separate until the network has selected among them, or until the encoding itself has recorded which is which, and pool wherever selection is unnecessary.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal fusion, multimodal learning, early fusion, binding problem, additive fusion, identifiability, linear attention, linear recurrent models, delta rule, remote sensing, weather forecasting, synthetic benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_57PEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposed a Tokaformer that can preserve typed tokens by a linear-recurrent routing before compression after analyzing the information loss in shared additive fusion and theoretically showing its inability to preserve track information. They conduct experiments to support the benefits of identity-preserving fusion and the effectiveness of their method.

Critical strengths
  1. They offered a clear theoretical analysis of the information loss in shared additive fusion and its inability to preserve track information, which motivates their Tokaformer natually and smoothly. The method specifically design related mechanism to solve the problem.

  2. They did comprehensive experimental validation by conducting experiments across multiple synthetic tasks and two real-world datasets and ablation studies to support their claims.

Critical weaknesses
  1. For their proposed method, the idea is highly similar to prior work (paper name: [Climax: A foundation model for weather and climate]), which also preserves specific representations and performs cross interaction before compression. This means the novelty of this work is not enough.

  2. Most of improvement are shown on on synthetic tasks while the real-world results show limited advantages for Tokaformer.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer 57PE

Thank you for the review.

W1, similarity to ClimaX. Please see G1. ClimaX's cross-attention pools the variables at each spatial position into one token, and that pooling is the compression step; no interaction across positions happens before it. Tokaformer's routing stage operates across positions while typed tokens are still separate. We have added ClimaX's variable aggregation as a baseline (G1). With its own tuned recipe and no residual it trains, scoring 1.000 on the panel's positive-control cell, but on the task that requires moving information between positions it scores 0.0001 normalised accuracy (chance; n = 30), against 0.79 for Tokaformer on the same data. We accept that the submitted text did not make this distinction clear; the revision states it in the introduction and adds the comparison as Table 2.

W2, synthetic versus real-world evidence. Please see G2. The real-world experiments test the paper's prediction about encoder dependence, and both datasets are consistent with it. The revision presents them as a test of that prediction, and the limitations state that neither dataset measures cross-position retrieval. G2 names real tasks that do; we have not run them.

Presentation. The revision states the distinction of G1 in the introduction, adds the theorem's role to Section 2, and moves the baseline comparison into the main text as Table 2.

Thanks for your claims and explanation. However, I still have concerns about the novelty and effectiveness of your work. I will maintain my original score.

Reviewer_THSYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Tokaformer, a multimodal architecture that represents co-located modalities as separate typed tokens, processes them through a recurrent routing stage, and subsequently compresses the representation. The authors show theoretically that summing embeddings from a shared encoder with additive track identifiers loses information about which input produced which value. Experiments on synthetic retrieval and aggregation tasks, WeatherBench 2, and BigEarthNet demonstrate that separate tokens help when input identity is lost during fusion, while conventional summation can remain effective when the encoder already preserves this information.

Critical strengths
  1. The paper studies an interesting problem of information loss during early multimodal fusion. The theoretical analysis clearly characterizes when shared additive encoding loses track-value associations, while the disjoint-encoding control demonstrates that this limitation can be avoided without abandoning summation itself.

  2. The experimental analysis is relatively comprehensive, with useful controls including length-matched pause tokens, independent hyperparameter tuning, routing-stage placement, and aggregation tasks. The real-world evaluations on WeatherBench 2 and BigEarthNet further demonstrate how the relative benefits of fusion strategies depend on the underlying input encoder.

Critical weaknesses
  1. The practical advantages over existing binding-preserving fusion approaches remain unclear. The theoretical limitation applies specifically to shared encoders with additive track offsets, while existing methods such as ClimaX (ICML 2023) and Perceiver (ICML 2021) already preserve input identity before compression. In Figure 4, per-channel summed fusion substantially outperforms factored fusion on WeatherBench 2 (89.4 vs. 150.3 RMSE) and performs similarly on BigEarthNet (0.710 vs. 0.708 macro-AP). Thus, the benefits of the proposed method seem primarily demonstrated against a restricted shared-encoder baseline, rather than stronger conventional fusion architectures.

  2. The advantage over alternative attention-based routing methods is not sufficiently established. The authors evaluate primary-query cross-attention in Appendix D, but find no robust advantage for Tokaformer at shorter sequence lengths. While longer-sequence experiments favor recurrent routing in training reliability, the comparisons primarily use hyperparameters selected at shorter lengths, and the additional tuning experiments do not provide a completed held-out comparison. A stronger accuracy–compute comparison against properly tuned cross-attention baselines would help establish the benefit of the proposed architecture.

  3. (Relatively less critical) The hybrid fusion strategy relies on a manually specified partition of inputs that can be pooled versus retained separately. While it reduces routing tokens by 35% with similar accuracy on synthetic tasks, it is unclear how this partition could be determined automatically or how well the efficiency benefits transfer to real-world settings.

Questions for authors

Please see weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer THSY

Thank you for the clear summary and for separating the more and less critical concerns.

W1, advantage over binding-preserving approaches. Please see G1 and G2. We agree the theorem applies to shared encoders with additive offsets. Our claim about ClimaX-style and other identity-preserving fusions is different and is tested separately: they pool to one token per position before positions interact, and on the cross-position task every such baseline in our panel scores at most 0.002 against 0.79. On Figure 4, the per-channel condition is the one in which summation is lossless (the trained input map has rank 69), so the analysis predicts no gain from separate tokens there, and summed fusion is the better choice; under a shared encoder, where the theorem applies, Tokaformer is better on both datasets (146.0 vs 249.2 RMSE; 0.714 vs 0.695 macro-AP). G2 names real tasks that need cross-position retrieval; we have not run them.

W2, comparison with attention-based routing. We agree with this assessment of Appendix D. There is no robust advantage at short lengths, and the main text says so (Section 5). The long-length result is a reliability result under frozen recipes (15/15 vs 6/15 at L = 1024; 15/15 vs 1/15 at L = 2048), not an accuracy–compute frontier. A recipe search for cross-attention optimised at L = 2048 reached the success bar in 1 of 48 runs (best configuration 0.29 on average), and a larger development search gave 5 of 48 at L = 1024 and 3 of 48 at L = 2048, against 48 of 48 for Tokaformer at both lengths (Appendix D). These are outcomes of bounded tuning, not an impossibility result. The revised limitations name the missing accuracy-versus-compute comparison.

W3, hybrid partition. For WeatherBench the partition was not tuned by hand. It was fixed before training from a data-side statistic: each field's novelty, defined as 1 − R² when the field is predicted from the other fields at the same grid cell. Low-novelty fields (the mid-level geopotentials Z150 to Z600) were pooled and the other 62 kept as typed tokens. With that frozen partition the hybrid improves on full factoring at equal wall-clock (137.4 against 150.3 RMSE), though it stays behind summed fusion in this condition, for the reason given in G2. The revision describes the procedure (Section 4.1, Appendix E). A learned partition is future work and is listed as a limitation.

Thank you for the clarifications. From the experimental side, I believe the paper can be strengthened by adding more real-world evaluations against existing fusion baselines (e.g. further cross-position retrieval comparisons). I don't have sufficient theoretical expertise to gauge the theoretical contributions and will defer to other reviewers on that. I retain my original score.

Reviewer_LHkhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes TokaFormer, a multimodal architecture that preserves modality-specific information before compressing inputs into a shared representation. The main motivation is that early additive fusion using a shared encoder can lose the association between a modality and its corresponding value, making selective retrieval difficult. The authors formalize this limitation through a theorem and propose keeping co-located modalities as separate typed tokens, followed by a recurrent routing stage before compressing them into primary tokens. The approach is evaluated on synthetic tasks involving selective retrieval, routing, and aggregation, along with WeatherBench 2 and BigEarthNet. The results show strong improvements in selective retrieval tasks, although the gains on real-world datasets depend significantly on how modality information is encoded.

Critical strengths

The paper addresses an interesting problem in multimodal fusion, particularly understanding when early compression loses information needed for downstream tasks. I find the distinction between selective retrieval and aggregation useful, as it helps explain when modalities need to be preserved separately.

The theoretical analysis provides a clear explanation of information loss under shared additive encoding. The disjoint-encoding control helps establish that the limitation comes from how modality identity is represented rather than addition itself.

The synthetic experiments are fairly comprehensive and well controlled. In particular, the pause-token experiments and recurrent-stage placement ablations help demonstrate that the improvements are not simply due to additional tokens or computation.

The hybrid fusion approach is interesting, showing that not all modalities need to remain separate throughout the network. The approach achieves comparable performance to full factoring with around 35% fewer routing tokens, suggesting a useful direction for more efficient multimodal representations.

Critical weaknesses

My main concern is the novelty and scope of the theoretical contribution. The theorem establishes that a shared encoder with additive modality offsets loses the association between modality identity and its value after summation. While mathematically correct, this follows directly from the permutation invariance of addition and applies to a relatively restrictive encoding setup. Existing approaches using modality-specific encoders or role-value representations already avoid this limitation. It is therefore unclear how much new insight the theorem provides beyond formalizing this known issue.

The architectural novelty relative to existing multimodal fusion approaches is not fully established. Methods such as ClimaX already preserve variable-specific representations before applying cross-attention-based aggregation, while Perceiver uses learned attention to compress structured inputs. The paper does not convincingly demonstrate why this is preferable to existing identity-preserving fusion mechanisms.

The real-world results are mixed. On WeatherBench 2, summed fusion with modality-specific encoders substantially outperforms the proposed factored architecture (89.4 versus 150.3 RMSE). On BigEarthNet, the two approaches perform similarly under modality-specific encoders. These results are consistent with the authors' theoretical argument, but they also raise questions about the practical advantage of introducing additional routing tokens when the input encoder already preserves modality identity.

Most of the strong experimental evidence comes from synthetic tasks designed to expose the limitations of early shared additive fusion. A stronger evaluation against identity-preserving baselines, particularly on tasks requiring cross-position selective retrieval, would help establish the broader relevance of the approach.

The hybrid fusion strategy requires manually specifying which modalities can be aggregated and which should remain separate. In practice, this decision may depend on the task or downstream query. It is unclear how the architecture would make this distinction automatically in more general multimodal settings.

Questions for authors

What is the main architectural contribution over existing modality-preserving fusion approaches? Can the authors clarify what the recurrent routing stage provides beyond these existing approaches, particularly when modality identity is already preserved?

How should we interpret the real-world results where summed fusion performs as well as or better than TokaFormer?

Can the authors provide a more direct comparison against cross-attention-based fusion under matched computational budgets?How do the methods compare when training compute, model capacity, and inference cost are controlled, with appropriate tuning for each architecture?

How well does the proposed routing mechanism generalize to unseen input configurations? The synthetic experiments show some degradation when transferring across unseen contact geometries. Can the authors clarify whether the model learns a general routing mechanism or primarily learns the particular relationships present during training?

How would the hybrid fusion strategy generalize to settings where the relevant modalities are not known in advance. Could this partition be learned or made query-dependent, and how would that affect efficiency?

1 comment
AuthorsOpenReview ↗

Response to Reviewer LHkh

Thank you for the detailed review and the specific questions.

W1, theorem scope. Please see G3. We agree it formalises a simple fact, and that modality-specific encoders avoid it; the disjoint control in the paper is exactly that case. What we believe is new is the corollary and its test: avoiding the binding loss is not enough for tasks that need cross-position retrieval (G1).

W2 and Q1, contribution over ClimaX and Perceiver. Please see G1. In short: those methods preserve identity up to a per-position pooling step, and nothing mixes across positions before that step. Tokaformer's routing stage runs while typed tokens are still separate. When identity is already preserved, the routing stage is what provides cross-position transport; the position-local, identity-preserving baselines score at most 0.002 on the transport cell against 0.79.

W3 and Q2, real-world results. Please see G2. The per-channel condition is the one in which summation is lossless (the trained input map has rank 69), so the analysis predicts no gain from separate tokens there; the shared-encoder condition is the predicted separation, and it appears on both datasets.

W4, identity-preserving baselines on cross-position retrieval. The five position-local baselines in the panel are identity-preserving and are evaluated on exactly this kind of task; the revision moves this result from the appendix into the main text (Section 3.2, Table 2) and adds a ClimaX-style arm. It solves the panel's positive-control cell (1.000) and is at chance on the transport cell (0.0001 against 0.79, n = 30; G1).

Q3, cross-attention under matched budgets. At short sequence lengths we find no robust advantage over primary-query cross-attention, and the paper says so (Section 5 and Appendix D). At longer lengths, with recipes frozen in advance, Tokaformer trains successfully in 15/15 runs at L = 1024 and 15/15 at L = 2048, against 6/15 and 1/15 for cross-attention. To check that this is not a tuning artefact, we ran a separate recipe search for the cross-attention comparator at L = 2048, optimised for that length: 1 of 48 runs reached the 0.9 success bar, and the best configuration averaged 0.29. A later, larger development search gave each method twelve recipes per length at up to 6,000 updates: Tokaformer crossed the bar in 48 of 48 cells at both lengths, and cross-attention in 5 of 48 at L = 1024 and 3 of 48 at L = 2048. Both searches are in Appendix D, and the revision brings them into Section 5. We report them as outcomes of bounded tuning, not as an architectural impossibility, and no final comparison with frozen winners was run. We do not have a complete accuracy-versus-compute curve for both methods, and the revised limitations say so.

Q4, generalisation to unseen configurations. There are two different tests. (i) Recipe transfer: hyperparameters selected on development geometries, then models trained on five fresh geometries with three seeds each; the gap between Tokaformer and summed fusion is 1.00 (95% CI 0.999 to 1.006). The method does not depend on a particular geometry. (ii) Zero-shot model transfer (Section 3.1, Fig. 2k), which we take to be the degradation the reviewer refers to: a model trained on one contact map and evaluated on unseen maps. Here Tokaformer retains about 0.45 normalised accuracy at depths 2, 4 and 8, against 0.35, 0.07 and 0.00 for summed fusion. So the learned routing is partly general and partly specific to the training map: it transfers well above the summed baseline and does not fall with depth, but it is far from the in-distribution level. The paper says that transfer is modest for both methods and lists it under limitations; the revision gives these numbers in Section 3.1.

W5 and Q5, the hybrid partition. The submitted text says the partition is specified by hand; that is true of the synthetic task but not of the real data. For WeatherBench the partition was fixed before any hybrid model was trained, from a statistic that needs no trained network (a ridge regression on the training data): for each field we measure how well it is predicted from the other fields at the same grid cell (held-out R²), and call 1 − R² its novelty. Fields with low novelty add little that the pooled primary does not already carry, so they are summed; the rest stay as typed tokens. This pooled the mid-level geopotentials Z150 to Z600, which with the primary are the seven lowest-novelty fields of the 69 (0.34 to 0.60, against 1.0 for the stratospheric fields), and kept the other 62 fields separate. That frozen partition is the hybrid in Figure 4, which improves on full factoring at equal wall-clock (137.4 against 150.3 RMSE). The revision describes this in Section 4.1 and Appendix E. It is a redundancy heuristic computed once per dataset; we have not evaluated a learned or query-dependent partition, and the limitations say so.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3): changes and novelty over ClimaX and Perceiver

We thank the reviewers for careful and consistent reviews. Three concerns are shared: novelty relative to ClimaX and Perceiver, the scope of the theorem, and the real-world results. We answer them here and refer back from the individual responses. We have uploaded a revised PDF; the changes are:

  • Introduction and related work: the distinction between binding within a position and transport between positions, and how ClimaX and Perceiver sit with respect to it.
  • Section 3.2, new Table 2: identity-preserving fusions on the cross-position task, including a new ClimaX-style arm (details in Appendix C).
  • Section 2: a statement of what the theorem is for.
  • Section 5 and Figure 4: the real-data comparisons stated as a two-sided prediction, with the reason summed fusion wins under per-channel encoders (the theorem does not apply there, and separate tokens cost optimiser updates); the cross-attention retuning searches, previously only in the appendix; update counts in the caption.
  • Abstract and limitations: the transport result, and a statement that neither real dataset measures cross-position retrieval.
  • Section 3.1: the zero-shot transfer numbers.
  • Section 4.1 and Appendix E: how the hybrid partition was chosen for WeatherBench.

G1. What Tokaformer does that ClimaX and Perceiver do not

The reviews describe ClimaX as preserving variable-specific representations and then applying cross-attention before compression. In ClimaX the cross-attention is the compression. Its variable aggregation runs a single learned query over the V variable tokens at one spatial position and returns one token for that position. Positional embeddings and the transformer trunk come after. Variable tokens at position i never interact with tokens at position j while they are still separate.

The distinction we draw is therefore not whether modality identity is preserved at the input, but how many tokens per position exist at the moment positions exchange information:

MethodIdentity preserved at inputTokens per position when positions mixCross-position mixing before compression
Summed, shared encoderNo1No
Summed, per-modality encodersYes1No
ClimaX variable aggregationYes1No
PerceiverYesn/aNo: inputs are only read by latents; positions mix among the latents, after compression
TokaformerYes1 + number of modifier tokensYes

Our related-work section says that Perceiver and ClimaX "already bind before compressing" and that Tokaformer "differs in mechanism". That sentence is correct about binding, and we think it is the source of this concern: it does not say what the mechanism is for. Binding within a position and moving information between positions are separate requirements, and the submitted text does not separate them. The revision replaces the sentence with this distinction.

Our baseline panel (Appendix C, Fig. A2a) already contains five fusions that preserve identity but emit one token per position (gated sum, sum-then-MLP, concatenate-then-project, FiLM, low-rank bilinear). The submitted appendix called the panel's eight arms "position-local"; three are the summed, factored and dense-attention arms, and the revision corrects the wording. On the cell that requires moving information between positions (joint task, distance 4, depth 4 and 8; n = 30 per fusion), every one of them scores at most 0.002 normalised accuracy, against 0.79 for Tokaformer. Concatenate-then-project keeps every modality in its own feature block, so identity is fully preserved, and it still fails.

To address the ClimaX comparison directly, we added ClimaX's variable aggregation to this panel, following the reference implementation (per-variable embedding, zero-initialised learned query, multi-head cross-attention per position, no residual), on the same seeds and maps as the existing arms. We gave the arm its own twelve-recipe optimiser search on the panel's positive-control cell, where the answer is two positions away and nothing else is written at the source, and froze the best recipe before the transport run. On the control cell it scores 1.000 normalised accuracy (n = 15), so it trains and can read a neighbouring position. On the transport cell it scores 0.0001 (SD 0.006, n = 30), against 0.79 for Tokaformer on the same 30 data streams, with twice Tokaformer's training steps. Preserving variable identity up to a per-position pooling step does not provide transport between positions.

The search was needed: at the panel's default learning rate the arm does not train even on the control cell (0.0004). A variant that adds the summed embedding as a residual trains at the default rate (0.996 on the control) and gives the same answer on transport (0.0015). FiLM and concatenate-then-project carry that residual in the panel.

AuthorsOpenReview ↗

General response (2/3): reading the real-world results

G2. Reading the real-world results

The real-data experiments are a two-sided test of the analysis, not a benchmark of the architecture. The analysis predicts where separate tokens help and where they cannot. Both datasets go the predicted way, and on WeatherBench the per-channel result is stronger than "no gain": summed fusion is clearly better.

ConditionPredictionWeatherBench 2, z500 RMSE (lower is better, n = 5)BigEarthNet, macro-AP (higher is better, n = 3)
Shared encoder: track identity is lost in the sumSeparate tokens winTokaformer 146.0 ± 4.7, summed 249.2 ± 8.8Tokaformer 0.714 ± 0.004, summed 0.695 ± 0.001
Per-channel encoders: identity survives the sumNo information gain from separate tokens; their cost remainsTokaformer 150.3 ± 3.6, summed 89.4 ± 1.0Tokaformer 0.708 ± 0.002, summed 0.710 ± 0.003

Where the prediction could have failed. Under a shared encoder the theorem applies, and summed fusion should degrade while Tokaformer should not. That is what happens on both datasets: Tokaformer is close to insensitive to the encoder, and summed fusion's WeatherBench error nearly triples. More updates do not close the gap within the budget: summed fusion takes about 12× more updates than Tokaformer there and is still worse, and its first checkpoint is already about 2.8× worse than Tokaformer's last (Fig. 4d). A shared encoder is not an artificial restriction. Models built for varying channel sets use one: ChannelViT (Bao et al., 2024) and LaBraM (Jiang et al., 2024) pass every channel through a shared encoder and add a channel embedding. They then keep one token per channel at full attention cost. Our shared-encoder summed arm is the cheaper alternative, and Figure 4 shows what it loses.

Why summed fusion wins with per-channel encoders. Per-channel encoders already keep the fields apart: on WeatherBench the model is 128 wide, there are 69 fields, and the trained input map has rank 69 (Appendix E; now stated in Section 5). The theorem does not apply, and what is left is the cost of separate tokens: summed fusion completes about 16× more optimiser updates in the same wall-clock time. We agree with the reviewers' reading: in the standard per-channel configuration on WeatherBench, Tokaformer is clearly worse, not level. The submitted text did not say that they are the predicted outcome, and the revision does.

Scope. Neither benchmark requires retrieving one specific input's value at one position from another position, and the revised limitations say so. That case is tested directly in Table 2 (G1). Real tasks with this structure exist, for example filling a cloud-covered optical band from the same band at a cloud-free date, or imputing one sensor at a station from the same sensor at neighbouring stations. We have not run them.

AuthorsOpenReview ↗

General response (3/3): what the theorem contributes

G3. What the theorem contributes

We agree the proof is short. Its role in the paper is to locate the failure and make testable predictions:

  1. It attributes the loss to shared encoding, not to addition. The disjoint-encoding control keeps summation and restores accuracy (0.054 / 0.103 / 0.123 for shared-sum at K = 2 / 4 / 8, against 1.000 / 0.998 / 0.996 for disjoint-sum).
  2. It separates an information limit from a training difficulty. On the task that instantiates the theorem, the shared sum reaches 0.527 / 0.327 / 0.233 raw accuracy at K = 2 / 4 / 8, against ceilings of 0.531 / 0.337 / 0.258 derived from the theorem: it learns nearly everything the fused vector contains, and no amount of width or depth can add more. On the position-addressed task the summed input is identifiable in principle, and there the failure is one of trainability that moves with the training budget. We fixed in advance a prediction that it would scale with width and embedding rank; that prediction failed, and we report it (Appendix C, Fig. A1). The paper keeps these two cases apart throughout, and the theorem is what lets us do so.
  3. It identifies which real-world configuration fails. The theorem's setting is a shared encoder with additive type embeddings. That is the configuration in which summed fusion fails on both datasets (G2), and the per-channel configuration, where the theorem does not apply, is the one in which it does well.
  4. It raises the question the reviewers asked. Preserving identity is necessary; is it sufficient? A fusion that emits one token per position does nothing itself to move information between positions, however it combines modalities within a position. G1 tests this: at the training budget of our panel, every such fusion stays at chance on the transport cell.

The revised Section 2 states this role directly after the proof.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.