acceptodds
Under review as a conference paper at ICLR 2027

RandSlot: Learning Compact Visual Document Representations with Random Soft Tokens

13 views

Abstract

Visual document retrieval requires expressive representations to match queries with evidence distributed across text, tables, and page layouts. Multi-vector representations capture fine-grained information, but storing and comparing many vectors introduces substantial retrieval costs. In this paper, we introduce RandSlot, a simple approach to learn compact visual document representations with random soft tokens. During training, we append independently-sampled random unit vectors to query and document input sequences and resample them at every use, without introducing learnable soft-token parameters. The encoder contextualizes these auxiliary inputs with the original content to produce a small set of retrieval vectors. A standard late-interaction objective trains the encoder to extract relevant information under varying input conditions. Experiments with different backbone models show that RandSlot improves retrieval quality over alternative readout strategies under the same vector budget. Further analysis shows that these gains can persist when random soft tokens are replaced with zeros at inference, demonstrating that random inputs during training can improve compact retrieval representations even when inference no longer requires sampling.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual Document Retrieval, Compact Multi-Vector Representations, Random Soft Tokens, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ucqmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce RandSlot, a parameter-free readout strategy for learning compact multi-vector representations in visual document retrieval. Instead of using learnable latent tokens (e.g., MetaEmbed) or feeding model-generated representations back autoregressively (e.g., CausalEmbed), RandSlot appends K−1K-1 independently sampled, L2-normalized Gaussian random vectors to the query and document sequences during every encoding call. A shared causal vision–language encoder (adapted via LoRA) extracts KK L2-normalized hidden states, one from the terminal token of the original content and K−1K-1 from the appended random positions, which are jointly optimized using a directional MaxSim InfoNCE objective.

Critical strengths

The central finding that continually resampled random inputs outperform persistent learnable tokens, autoregressive feedback, and fixed/zero tokens, and still transfer to deterministic zero-input inference is both practical and conceptually illuminating.

Critical weaknesses
  • Unlike Direct-Feedback (where input ete_t depends on output rtr_t), RandSlot’s auxiliary inputs e1,…,eK−1e_1, \dots, e_{K-1} are sampled independently of the model's outputs. In a causal transformer, feeding the entire extended sequence [Xx;e1;… ;eK−1][X_x; e_1; \dots; e_{K-1}] in a single parallel forward pass produces the exact same hidden states at the final KK positions as feeding prefixes sequentially. However, Appendix A ("Readout execution") states that training recomputes the full prefix KK times sequentially and inference decodes step-by-step with KV caching. This sequential implementation inflates training cost and causes an artificial 47.4% jump in query encoding latency (from 109.92 ms to 162.08 ms in Table 5) when scaling from K=4K=4 to K=6K=6. Simultaneously, Table 13 reveals that on the query side, using a single readout alone (r3r_3: 45.42%) outperforms the full 4-vector query set, and removing r1r_1 or r2r_2 improves retrieval. Thus, RandSlot fails to induce complementarity on the query side, meaning 75% of the query budget and MaxSim dot products at test time are redundant.
  • In the controlled ablations, Learned-Feedback (adapted from MetaEmbed) and Direct-Feedback (adapted from CausalEmbed) are implemented without their original method-specific training objectives (i.e., omitting MetaEmbed's Matryoshka nested-budget loss and CausalEmbed's auxiliary losses). Because those auxiliary objectives were specifically designed to regularize multi-vector generation, omitting them likely exacerbates the severe slot collapse observed for L and DF in Figure 4, making the controlled comparison less charitable to those paradigms. Moreover, all models are trained exclusively at K=4K=4, where inference scaling saturates at K=6K=6 (46.0% on 3B) and leaves a notable 4.2-point gap to the full-token Base model (50.2% on ViDoRe V3).
Questions for authors
  1. Since e1,…,eK−1e_1, \dots, e_{K-1} are known prior to the forward pass in RandSlot (as well as in Z, F, and L), why did you recompute the full prefix sequentially during training and run sequential KV-cached steps at inference instead of passing all K−1K-1 soft tokens in a single parallel forward pass? Is there a specific architectural constraint in Qwen2.5-VL / Qwen3-VL (such as M-RoPE position ID assignment) that prevents single-pass extraction of r1,…,rKr_1, \dots, r_K, or is the 47.4% latency increase from K=4K=4 to K=6K=6 in Table 5 purely an implementation artifact?
  2. Table 13 demonstrates that a single query vector (r3r_3) matched against 4 document vectors (Kq=1,Kd=4K_q=1, K_d=4) achieves 45.42% on ViDoRe V3—matching or slightly exceeding the default 4/44/4 configuration while reducing MaxSim dot products from 16 to 4 and eliminating auxiliary query steps. Why does directional MaxSim fail to produce complementary query readouts under RandSlot, and does this single-query-vector sufficiency also hold on Qwen3-VL-4B?
  3. Given that Learned-Feedback and Direct-Feedback exhibit extreme slot collapse in Figure 4 (H=0.410H=0.410 and H=0.150H=0.150), to what extent is their underperformance in Table 3 caused by stripping away MetaEmbed’s nested-budget loss and CausalEmbed’s auxiliary losses? Did you experiment with adding nested-budget (Matryoshka) supervision to Learned-Feedback or RandSlot, which might also resolve the performance drop observed at K=1K=1 and K=2K=2?
6 comments

Q1: Since e1,…,eK−1e_1, \dots, e_{K-1} are known prior to the forward pass in RandSlot (as well as in Z, F, and L), why did you recompute the full prefix sequentially during training and run sequential KV-cached steps at inference instead of passing all K−1K-1 soft tokens in a single parallel forward pass? Is there a specific architectural constraint in Qwen2.5-VL / Qwen3-VL (such as M-RoPE position ID assignment) that prevents single-pass extraction of r1,…,rKr_1, \dots, r_K, or is the 47.4% latency increase from K=4K=4 to K=6K=6 in Table 5 purely an implementation artifact?

One-pass inference resolves the serial-execution issue. All auxiliary inputs are available in advance. Our Qwen2.5-VL-3B and Qwen3-VL-4B implementations now process them in a single full-sequence forward pass, preserving the evaluator’s masks and position/M-RoPE convention. We compare the same checkpoints and pre-sampled inputs against sequential inference with KV caching.

Full-V3 quality and A100-80GB query encoding latency (Table 7; Appendix F.4, Table 25). S/O denotes sequential KV / one-pass:

Model/KnDCG@5 S/OQuery ms S/O
3B/445.45/45.41107.28/25.71
3B/645.80/45.80157.09/25.26
4B/446.09/46.06135.14/35.61
4B/646.27/46.29198.84/35.16

All macro-average changes are below 0.044 percentage points. For 3B, query encoding is 4.17×/6.22× faster at K=4/6, while page throughput increases from 4.07/3.35 to 6.05/6.05 pages/s. Thus, M-RoPE does not prevent this optimization, and the substantial K-dependent serial overhead is avoidable.

These BF16 measurements use batch size 1 with 32 query or 1536 visual tokens, excluding preprocessing, sampling, and retrieval. They establish nearly unchanged macro retrieval quality, not identical vectors or rankings.

For training, we retained sequential prefix recomputation primarily to keep the training procedure consistent with Direct Feedback for a controlled comparison of auxiliary-input schemes. This was an experimental-design choice, not an architectural constraint; one-pass training with dropout remains unvalidated. We will update the cost discussion accordingly.

Q2: Table 13 demonstrates that a single query vector (r3r_3) matched against 4 document vectors (Kq=1,Kd=4K_q=1, K_d=4) achieves 45.42% on ViDoRe V3—matching or slightly exceeding the default 4/44/4 configuration while reducing MaxSim dot products from 16 to 4 and eliminating auxiliary query steps. Why does directional MaxSim fail to produce complementary query readouts under RandSlot, and does this single-query-vector sufficiency also hold on Qwen3-VL-4B?

Turning query redundancy into a matching tradeoff. In the original 3B diagnostic run, retaining query readout rq,3r_{q,3} with four document vectors scores 45.42, versus 45.26 with all four query vectors. The score for one query vector and four document vectors is

s(q,d)=max⁡j=1,…,4rq,3⊤rd,j,s(q,d)=\max_{j=1,\ldots,4} r_{q,3}^{\top} r_{d,j},

requiring four DD-dimensional dot products instead of 16. This identifies an opportunity to cut matching cost fourfold while retaining the diagnostic ranking quality (Fig. 4 and Table 15, Appendix D).

Encoding and matching savings have different sources: one-pass inference removes serial model calls, while selecting rq,3r_{q,3} reduces matching vectors. Obtaining rq,3r_{q,3} still requires preceding causal positions. The original 3B selection is post hoc and needs independent deployment validation.

As a cross-backbone check, the newly reported 4B query-readout rq,3r_{q,3} scores 46.117 versus 46.201 for the full 4/4 representation; the post-hoc best query position, rq,4r_{q,4}, scores 46.297 (Table 6; Appendix F.3, Table 23). These within-run matching restrictions do not establish identical rankings or statistical equivalence.

Directional MaxSim allows multiple query readouts to select the same document position, so query redundancy is compatible with the objective; our complementarity claim concerns document readouts.

Q3: Given that Learned-Feedback and Direct-Feedback exhibit extreme slot collapse in Figure 4 (H=0.410H=0.410 and H=0.150H=0.150), to what extent is their underperformance in Table 3 caused by stripping away MetaEmbed’s nested-budget loss and CausalEmbed’s auxiliary losses? Did you experiment with adding nested-budget (Matryoshka) supervision to Learned-Feedback or RandSlot, which might also resolve the performance drop observed at K=1K=1 and K=2K=2?

What restored objectives change. We added DF with the document/query auxiliary terms and Learned with the average of the 1/1, 2/2, and 4/4 retrieval losses. These controls retain our original 3B adaptation recipe and four-readout training budget.

Retrieval performance. Under this recipe, DF scores 42.1 without the auxiliary terms and 42.23 with them, while Learned scores 42.9 without nested supervision and 41.71 with it (Table 3(A/C)). At the reported precision, these differences are approximately +0.13 pp and -1.19 pp, respectively. RandSlot scores 45.4 under the common recipe. Thus, restoring these objectives does not close the observed retrieval gap in these single-seed runs. These results do not establish a general effect of either objective across training seeds or recipes.

Readout utilization. Across the same 74,016 positive pairs, H is normalized marginal winner entropy and U is the mean number of distinct document winners per pair (Fig. 6 and Table 22, Appendix F.2):

MethodHU
Original DF0.1501.066
DF + auxiliary0.3951.358
Original Learned0.4101.361
Learned + nested0.6011.927
RandSlot0.9091.376

Both added objectives alleviate concentration. RandSlot retains the highest global winner entropy, while Learned+nested uses the most distinct winners within a pair. These utilization improvements do not necessarily translate into higher retrieval scores, so we distinguish readout utilization from retrieval quality.

Low-budget behavior. Table 17 (Appendix E.2) reports 37.6264/39.5986/41.7106 for the same Learned+nested checkpoint at 1/1, 2/2, and 4/4. Historical RandSlot/Zero-Feedback scores are reference points, not paired reruns. Without matched, verified low-budget evaluations of Learned without nested supervision, these results do not isolate the effect of nested supervision at 1/1 or 2/2.

Separate released-code comparison. We also retained the released CausalEmbed objective and auxiliary losses in a matched, single-seed 4/4 input-replacement experiment. Replacing the original inputs with random inputs improves the score from 29.25 to 30.15 (+0.90 pp), with gains on seven of eight tasks (Appendix E.3, Table 18). This uses a separate recipe from Table 3 and provides additional retrieval evidence for the input-strategy benefit when auxiliary losses are retained.

Reviewer_T7QJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the cost of multi-vector visual document retrieval by learning a small, fixed set of KK vectors per query and page. The key insight is that you can learn a compressed representation in KK random inputs during training. These vectors, called random soft tokens, then are appended to the original input and get contextualized by the encoder by attending to previous, original, ones. Experiments over different backbones show that the method obtains the best retrieval performance at a fixed vector budget when compared to other vectors compression approaches. Additionally, they show that the method allow retaining most performance while reducing the storage cost significantly compared to vanilla multi-vector retrieval. Finally they highlight that the random vectors can be swapped by zero vectors at test-time and still perform well, even outperforming training with zero vectors, showing the benefits of randomness on robustness against input.

Critical strengths
  • Practicality of the method and novelty in the domain of application. The paper proposes a parameter-free approach to reduce the cost of multi-vector retrieval for VDR. The method is compelling as it retains most of the retrieval performance while substantially reducing storage and matching costs, without modifying the base model architecture. In a controlled setting, it outperforms all other auxiliary-input strategies tested, and to the best of my knowledge, training with random auxiliary inputs has not been explored in VDR.
  • Analysis beyond benchmark scores. The paper examines how document readouts are used during matching and what each contributes. With RandSlot, individual document readouts are weaker than with the controls, but they are used more complementarily. The analysis also reveals that query readouts are largely redundant across all input strategies, a general observation that motivates asymmetric query/document budgets in future work. The short analysis on cost (Section 5.3) is also valuable for practitioners
  • Reproducibility. The paper provides all the details to be reproduced, and only uses open-source datasets and models.
Critical weaknesses
  1. Comparison protocol. The main comparison (Tables 1 and 2) reuses published numbers from [3]. This is transparent, but the comparison is not controlled: CausalEmbed shares the backbone and training data, but differs in almost all other training choices (batch size, loss, learning rate, LoRA rank, temperature, ...). The gains over these rows therefore cannot be attributed to the method alone. The controlled comparison of input strategies (Table 3) is the meaningful one and should be emphasized in the abstract and introduction instead.
  2. Analysis of the method performance. The analysis in Section 5.2 shows what changes (document readouts are used more complementarily), but not why training with random inputs produces this behavior. Several explanations are plausible (regularization, invariance to slot inputs, avoiding position-specific shortcuts), but none is tested directly. The observation that random inputs can be replaced by zeros at inference also deserves discussion. Such analysis would increase confidence in the method and suggest directions for improvement. Part of the experiment section, which mostly lists results without takeaways, could be traded for it.
  3. Missing baselines. A common way to improve multi-vector retrieval is to append special tokens from the existing vocabulary to the input (e.g., [MASK] tokens in ColBERT [1], padding tokens in ColPali [2]). This is also parameter-free, and its vector budget can be matched by reading the representation from the appended tokens only. Comparing against it would isolate the benefit of randomness from that of extra readout positions. Similarly, the Learned-Feedback baseline covers position-specific learned tokens, but not a single new vocabulary token repeated at each slot, which costs only ~2k parameters (one embedding row) and is therefore not a meaningful overhead.
  4. Compactness claim. The paper claims to learn compact visual document representations, but compactness is mostly discussed through vector counts, and costs are only analyzed in the short Section 5.3. Each vector keeps the full hidden dimension, which varies between backbone models (2,048–2,560). Making the cost axis more central in the discussion, with a fair metric, would strengthen the paper.
  5. Method definition and notation. Figure 1 conveys the method well conceptually, but the formal description is not straightforward. The input sequence XxX_x is never given an explicit form or shape, so the concatenation in Eq. 2 is ambiguous. Notation such as IDinI_{D_{in}} and N(⋅)N(\cdot) should be defined. The authors should also state that normalizing a Gaussian sample amounts to sampling uniformly on the unit sphere, and justify the choice of unit norm relative to the scale of real token embeddings. Finally, several terms are undefined or ambiguous, notably “readouts” and inputs that “carry content”. Addressing these would make the paper noticeably easier to follow.

Sources

[1] ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, Khattab et al., 2020

[2] ColPali: Efficient Document Retrieval with Vision Language Models, Faysse et al., 2024

[3] CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding, Huo et al., 2026

Questions for authors

See weaknesses.

6 comments
  1. Comparison protocol. The main comparison (Tables 1 and 2) reuses published numbers from [3]. This is transparent, but the comparison is not controlled: CausalEmbed shares the backbone and training data, but differs in almost all other training choices (batch size, loss, learning rate, LoRA rank, temperature, ...). The gains over these rows therefore cannot be attributed to the method alone. The controlled comparison of input strategies (Table 3) is the meaningful one and should be emphasized in the abstract and introduction instead.

Comparison protocol. Table 3 is our controlled input-strategy comparison: variants share the backbone, training recipe, and 4/4 output budget. The central three-training-seed result is RandSlot 45.03 +/- 0.40 versus Zero-Feedback 43.50 +/- 0.43. We will foreground this evidence in the abstract/introduction and label L/DF as input-mechanism controls, not full reproductions of MetaEmbed/CausalEmbed.

We also ran a single-seed 4/4 input-replacement comparison using released CausalEmbed code and retaining its auxiliary losses. With matched local settings, original/random inputs score 29.25/30.15 (+0.90 pp; seven of eight tasks improve; Appendix E.3, Table 18). This single-seed test uses a recipe separate from Table 3; it is not a full reproduction of the published protocol. Tables 1/2 remain published-system comparisons with different recipes; their gains cannot be attributed solely to input construction. Our 16/32 random-input run scores 40.96 versus the published 42.6, without a matched local original-input control. We will report this limitation and document the paired configurations.

  1. Analysis of the method performance. The analysis in Section 5.2 shows what changes (document readouts are used more complementarily), but not why training with random inputs produces this behavior. Several explanations are plausible (regularization, invariance to slot inputs, avoiding position-specific shortcuts), but none is tested directly. The observation that random inputs can be replaced by zeros at inference also deserves discussion. Such analysis would increase confidence in the method and suggest directions for improvement. Part of the experiment section, which mostly lists results without takeaways, could be traded for it.

Analysis and zero-input inference. We directly tested auxiliary-input dependence on frozen 3B checkpoints, fixing content, positions, masks, and the 4/4 budget. Eight fixed diagnostic pools contain 16 queries/task (128 total) and 80-132 pages/task. Five shared Gaussian unit-vector banks, independent per sample, side, and position, give task-macro diagnostics (Table 5; Appendix F.1):

MethodQuery driftDocument driftTop-10 overlap (%)
Zero-Feedback0.13690.100678.47
Fixed-Random0.13440.103072.74
RandSlot0.01270.007394.25

Drift is mean 1-cosine between appended readouts across input draws. Z/F receive non-native random inputs as stress interventions; these sampled-pool diagnostics do not replace full-benchmark results.

For RandSlot, zero-minus-random diagnostic nDCG@5 is -0.15 pp (paired 95% CI [-0.93, 0.68]). The observed mean change is small on these pools; the interval does not establish equivalence. This query-bootstrap CI is conditional on checkpoints, pools, and five input draws, excluding training-seed uncertainty. Zeroing preserves readout positions and access to preceding content. Together, the interventions support reduced dependence on tested auxiliary-input values and help explain RS-Z. They do not identify a unique regularization mechanism, establish avoidance of positional shortcuts, or causally explain complementarity. The corresponding analysis is in Section 5.1 and Appendix F.1 (Table 21); we will shorten repetitive result descriptions.

  1. Missing baselines. A common way to improve multi-vector retrieval is to append special tokens from the existing vocabulary to the input (e.g., [MASK] tokens in ColBERT [1], padding tokens in ColPali [2]). This is also parameter-free, and its vector budget can be matched by reading the representation from the appended tokens only. Comparing against it would isolate the benefit of randomness from that of extra readout positions. Similarly, the Learned-Feedback baseline covers position-specific learned tokens, but not a single new vocabulary token repeated at each slot, which costs only ~2k parameters (one embedding row) and is therefore not a meaningful overhead.

Missing baselines. We added the two requested alternatives under the original 3B/4/4 recipe: a frozen existing special-token embedding repeated across appended positions, and one shared trainable vector initialized from it. The latter adds 2,048 parameters shared across positions and modalities. Their V3 scores are 41.98 and 43.01, versus RandSlot's 45.4 (Table 3(A/B)). All retain the original-content readout plus three appended readouts. Both token controls retain native input scale; this tests those alternatives as constructed, rather than isolating norm alone. We will add the results and construction details to the controlled comparisons.

Reviewer_TCoNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an approach of using a fixed number of random vectors appended to the input (document/query) being encoded into a set of output representations that are optimized for retrieval tasks. The key novelty here is the use of a very few number of independently randomly sampled vectors being able to encode auxiliary details of the input compared to the existing methods which require much larger number of token representations. Despite the smaller number of tokens, the method is able to show great improvement over SOTA which use larger token sizes.

Critical strengths
  1. The approach presented is a clean and simple mechanism to improve the encoding distribution of the embedding tokens for enabling more accurate retrieval.
  2. The methodology is explained well and the experiments conducted with appropriate controls, especially with the zero, fixed-random, learned, random sampled and direct feedback variants with the same token budget.
  3. The analysis reveals that the method produces representations for the tokens where all of them are equally relied upon for matching as opposed to concentrating on a single vector.
  4. The contribution of a highly efficient vector representation approach for retrieval is remarkable given that the previous SOTA relied on an order of magnitude greater number of tokens and still not reaching the performance of RandSlot.
Critical weaknesses
  1. What is the reasoning behind choosing a random soft token? Why did you select a normal distribution for sampling random vectors? That is not discussed in the paper. An ablation on the stochastic input construction would have strengthened the claim.
  2. Why directional MaxSim? Were other explored - like Mean or some basic fusion layers incorporated?
  3. From the method section 3.3, it is unclear how zero inputs (RS-Z) is actually implemented in training since the method only describes RS-R. "RS-Z preserves the positions, attention structure, and output budget without retraining". There could be some details provided for RS-Z here rather than just mentioning it in a line.
  4. The model is trained with a batch size of 128, however this style of training is very similar to CLIP which uses very large batch sizes to achieve maximal discriminative performance (more diverse samples within the batch). I don't see any discussions around this and how the sensitivity to the samples in the batch affects the performance of the method.
  5. In the discussions, there is limited mention of why the random resampling is beneficial, compared to the fixed-random or zero input or direct-feedback. The discussions merely describe the data in the tables without much value addition to the reader regarding why resampling helps generate better representations (sections 4.3 and 4.4).
  6. Since the method is a stochastic approach - randomly sampled input vectors are used, the results need to show that the method is stable across different inference runs.

Nit:

  1. Readability of the results in the table could be enhanced. Best results could be in bold and if needed second best underlined.

Overall, while the paper presents a very clean and elegant idea of using fixed random tokens to enhance retrieval, the major critique I have in this paper is the lack of discussion regarding the stochastic nature of their inference method and that most of the experiments simply use a single run to validate the performance. Since random sampling is involved as part of the input, that will naturally affect the output distributions and the paper fails to measure / present that which weakens the overall method's claim. Despite the properly identified controls used in their experiments and ablations, the absence of the stochastic inference behavior validation is a big gap in the paper's contribution.

Questions for authors

In figure 3 it is mentioned that the model is trained with 4 vector budget but inference is swept over 1->8 vectors. This is interesting since the 5-8 vectors would not have been trained for the retrieval task. How do you explain the consistent performance here?

11 comments
  1. What is the reasoning behind choosing a random soft token? Why did you select a normal distribution for sampling random vectors? That is not discussed in the paper. An ablation on the stochastic input construction would have strengthened the claim.

Why random soft inputs and normalized Gaussian sampling? RandSlot draws fresh, content-independent auxiliary inputs for each training encoding, whereas Fixed-Random reuses a stored table. Normalizing isotropic Gaussian samples gives uniformly distributed directions on the unit sphere, providing a simple, parameter-free sampling rule. Normalization controls input magnitude across examples and positions without introducing learned auxiliary parameters.

Under the original 3B/4/4 recipe, the repeated frozen-token and single shared trainable-vector controls score 41.98 and 43.01, respectively, versus RandSlot's 45.4 (Table 3(A/B)). These token controls retain native input scale. Together with the zero-input inference result, these comparisons support a training benefit beyond simply adding readout positions or a shared learned vector.

We also tested inference-input replacements on a frozen 3B RandSlot checkpoint using eight fixed sampled V3 diagnostic pools (16 queries per task, 128 total, and 80–132 candidate pages per task). Relative to unit-norm Gaussian inputs, changing the Gaussian norm to 0.3 or 3 changes the sampled-pool macro nDCG@5 by at most 0.107 pp, while unit-norm Rademacher inputs differ by 0.157 pp (Appendix F.1, Table 19). Each nonzero condition uses five input draws. These sampled-pool diagnostics are separate from full-corpus V3 evaluation.

The observed small mean changes support robustness to the tested inference distributions and scales. They do not establish statistical equivalence, compare training-time sampling distributions, or demonstrate that Gaussian sampling and unit norm are optimal. Unit scale remains a design choice rather than a value calibrated to native embedding norms.

  1. Why directional MaxSim? Were other explored - like Mean or some basic fusion layers incorporated?

Directional MaxSim, s(q,d)=∑imax⁡jqi⊤djs(q,d)=\sum_i\max_j q_i^\top d_j, lets each query readout select document evidence and keeps the retrieval objective constant across input controls. It permits repeated selection of the same document readout, explaining why query complementarity is not enforced. Mean/fusion scoring and batch-size sweeps remain outside the present experiments.

  1. From the method section 3.3, it is unclear how zero inputs (RS-Z) is actually implemented in training since the method only describes RS-R. "RS-Z preserves the positions, attention structure, and output budget without retraining". There could be some details provided for RS-Z here rather than just mentioning it in a line.

Thank you for pointing out this ambiguity. RS-R and RS-Z use the same RandSlot checkpoint, trained with random inputs at the appended slot positions. They differ only at inference: RS-R uses random slot inputs, whereas RS-Z replaces these inputs with zero vectors. The model weights, slot positions, attention structure, and number of output embeddings remain unchanged; no retraining is required. The Z baseline uses zero slot inputs during both training and inference.

The Full-V3 nDCG@5 results are:

BackboneRS-RRS-ZZ
3B45.445.443.9
4B46.145.845.1

RS-Z and Z both use zero slot inputs at inference, but RS-Z achieves higher scores. This supports the benefit of training with random slot inputs even when inference uses deterministic zeros. RS-Z also provides a practical inference option that requires no random sampling.

We have revised Section 3.3 to make this distinction between training and inference explicit.

Reviewer_ijN4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RandSlot for compact visual document retrieval. It uses random soft tokens to obtain multiple representations for queries and documents, which are matched using MaxSim. Experiments on ViDoRe benchmarks show improvements over several baselines, supported by ablation studies.

Critical strengths
  1. The proposed method is simple and effective, requiring only minor modifications to the existing framework.
  2. RandSlot achieves impressive retrieval performance across multiple benchmarks, outperforming several existing methods.
  3. The method produces compact representations without requiring complex architectural changes or autoregressive feedback.
Critical weaknesses

Major Comments:

  1. Limited motivation and methodological novelty. The proposed method builds on several familiar ideas, including auxiliary input tokens, causal contextualization, and contrastive learning. While the results are promising, I find the methodological novelty somewhat limited. The authors should better explain why random soft tokens are particularly useful for compact visual document retrieval and what makes this approach different from existing methods.
  2. Missing ablation on query-side and document-side randomization. RandSlot uses random soft tokens for both queries and documents, but it is unclear whether randomization is needed on both sides. The analysis in Appendix D examines query-side redundancy and document-side complementarity, but does not directly answer this question. I would like to see an ablation where random soft tokens are applied only to queries or only to documents.
  3. Limited analysis of random soft tokens. Random soft tokens are central to the proposed method, but their effects are not fully understood. The authors compare random, fixed-random, and zero inputs, but I think more analysis is needed. For example, how sensitive is the method to the random distribution or input magnitude? Would adding random noise to fixed or learnable tokens produce similar results? These comparisons would help determine whether the improvements come from the specific design or simply from adding randomness during training.

Minor Comments:

  1. Comparison with CausalEmbed. Since RandSlot is closely related to CausalEmbed, I think a comparison with the complete CausalEmbed method would be useful. The Direct-Feedback baseline in Table 3 does not include the auxiliary losses used in the original method, so it is not a full reproduction. It would be helpful to compare the two methods under the same backbone and training settings.
Questions for authors

My main question is why randomly resampled soft tokens improve retrieval performance. Is the improvement mainly due to stochastic regularization, or does randomization help the model learn more complementary representations? I would also like to know whether randomization is equally important for queries and documents. Some further analysis or discussion of these points would be helpful.

7 comments
  1. Limited motivation and methodological novelty. The proposed method builds on several familiar ideas, including auxiliary input tokens, causal contextualization, and contrastive learning. While the results are promising, I find the methodological novelty somewhat limited. The authors should better explain why random soft tokens are particularly useful for compact visual document retrieval and what makes this approach different from existing methods.

The gain goes beyond extra readout positions. The central question is how to train a few readouts to aggregate evidence distributed across a document page. RandSlot uses fresh, content-independent auxiliary inputs; their contextualized outputs gather useful content through the encoder. No trainable auxiliary embeddings or output-dependent input generation are added. Under the common 3B/4/4 recipe, full-V3 nDCG@5 is 45.4, versus 43.9/43.8/42.9/42.1 for zero/fixed-random/learned/direct-feedback inputs. Across three training seeds, the central comparison is 45.03 ± 0.40 versus 43.50 ± 0.43 for Zero-Feedback (+1.53 pp). New repeated frozen-vocabulary-token and single shared trainable-vector controls, retaining native scale, score 41.98 and 43.01. These fixed-budget controls demonstrate the value of resampling. Causal encoding and contrastive learning are shared components. See Table 3(A/B) and Appendix B.2 of the revised manuscript.

  1. Missing ablation on query-side and document-side randomization. RandSlot uses random soft tokens for both queries and documents, but it is unclear whether randomization is needed on both sides. The analysis in Appendix D examines query-side redundancy and document-side complementarity, but does not directly answer this question. I would like to see an ablation where random soft tokens are applied only to queries or only to documents.

Query-/document-side input interventions at inference. We evaluate a frozen 3B RandSlot checkpoint on eight fixed sampled V3 diagnostic pools, with 16 queries per task (128 total) and 80–132 candidate pages per task. Content, readout positions, attention masks, and the 4/4 readout budget remain fixed. These are sampled-pool diagnostic scores, separate from the full-corpus V3 result of 45.4.

Writing each policy as query input/document input, random/random, random/zero, zero/random, and zero/zero score 79.668, 79.516, 79.538, and 79.518, respectively. The four mean scores span less than 0.153 pp, indicating low average sensitivity to zeroing either side's auxiliary inputs within these diagnostic pools (Appendix F.1, Table 20). Zeroing changes the auxiliary-input values while preserving the readout positions and their access to preceding content.

These results characterize the inference robustness of the randomly trained model.

  1. Limited analysis of random soft tokens. Random soft tokens are central to the proposed method, but their effects are not fully understood. The authors compare random, fixed-random, and zero inputs, but I think more analysis is needed. For example, how sensitive is the method to the random distribution or input magnitude? Would adding random noise to fixed or learnable tokens produce similar results? These comparisons would help determine whether the improvements come from the specific design or simply from adding randomness during training.

New distribution/scale tests support inference robustness. Normalized isotropic Gaussian draws provide a simple, parameter-free way to sample uniform unit-sphere directions. We tested alternative inference inputs on a frozen 3B RandSlot checkpoint using eight fixed sampled V3 diagnostic pools (16 queries per task, 128 total, and 80–132 candidate pages per task; Appendix F.1, Tables 19 and 21). The results below are sampled-pool diagnostics, separate from full-corpus V3 evaluation:

Inference inputnDCG@5Delta vs Gaussian-1 (pp)
Gaussian, norm 0.379.626-0.042
Gaussian, norm 179.6680
Gaussian, norm 379.561-0.107
Rademacher, norm 179.511-0.157
Zero79.518-0.150

Nonzero conditions use five draws; zero is evaluated once. Across a tenfold Gaussian norm range, the maximum macro-score change is about 0.107 pp; the discrete unit-norm Rademacher alternative differs by 0.157 pp. Conditional paired 95% confidence intervals for the differences all include zero (50,000 within-task query bootstrap replicates, averaging draws first). The observed robustness therefore extends beyond the exact unit-Gaussian inference rule within these pools. The requested query-only/document-only stochastic-training ablation, and fixed/learned-plus-noise training, remain open; these inference interventions characterize learned robustness, not which side requires randomization during training or a unique causal regularizer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.