- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
- Introduce a-of-B filter: score only documents matching >=a terms of the top B query terms (sorted by weights). a is chosen from a small set of query.
- Introduce Theorem 1: candidate-count lower/uper-bound from itersection totals
- Introduce Theorem 2: lower bound on Top-K survival
- Implement an engine that parallelize search within a query by scoring multiple doc tiles/blocks at once using multiple threads. The paper claims 1.47x-3.14x speed-up over Naive baseline implemented by the authors and 8–29× over baselines at 16 T/Q1 setting (1 query, 16 threads)
Critical strengths
-
The finding that increasing hit constraint can rapidly shrink the candidate set while still retain a good mount of top-k candidate is interesting.
-
The introduced filter is simple and easy to implement. It demonstrates a nice jump over the naive baseline.
-
Code is available at time of review.
Critical weaknesses
-
Unfair comparison: baselines (Seismic, BMP, PISA) are single-threaded per query. At 1 thread Seismic/BMP faster on MS MARCO. At 16T1Q (16 threads, 1 query), the proposed approach wins most likely just because it implements within-query parallelism while the baselines do not. Note that: under load, throughput-oriented serving engines typically run one query per core. Many engines implement single-threaded query processing algorithms by design choice, not because of incapability.
-
The comparison is done at one single operation point (k=1000 at high fidelity ~0.97-0.99), this make the comparison/conclusions biased to this one single data point. Suggest the authors to look at the Pareto-frontier curves to compare the methods at different efficiency/effectiveness trade-off points. One method performs well at k=1000 might not perform well at other top-k values.
Questions for authors
-
Q1: Seismic 16T/Q64 on Dev = 29.4 ms vs 4.29ms on TREC19, why is there a difference here?
-
Q2: Exhaustive Splade-v3 MRR@10 41.08 vs published ~40.2? Please explain
-
Q3: Please provide the comparison with Pareto-frontier (single-threaded and balanced T/Q)?
8 comments
Response to Weakness2 and Q3: Adding Pareto-frontier curves
We thank the reviewer for requesting Pareto-frontier comparisons. We have added a new subsection, "Efficiency–Quality Trade-off," and included the requested Pareto plots across five query sets in the revised manuscript.
For fairness, we follow the published Pareto evaluations of Seismic and BMP, using the same parameters and search ranges as their original experiments.
For MultiHit, we deliberately explore lower-quality operating points by fixing the original and truncating posting lists to 5–200 thousand entries, without applying Q1–Q3 to retune the parameters.
The new plots show that MultiHit achieves higher ranking quality at low latency across all five query sets, outperforming Seismic and BMP in the low-latency region.
Although the result seems good, this comparison is not entirely favorable to MultiHit. Posting-list truncation greatly changes the underlying data distributions, so the original is not necessarily optimal for these new settings. We intentionally avoid testset-specific retuning to maximize the Pareto frontier, since doing so would neither establish a generalizable parameter-selection rule nor further support our Q1–Q3 theory.
We emphasize that the requested Pareto-frontier comparison is an engineering evaluation, and is not related to our Q1--Q3 theoretical contributions. Our core MultiHit filter has only two parameters, , whereas Seismic provides 6 tuning options, including n_postings, centroid_fraction, summary_energy, query_cut, heap_factor, and n_knn, and BMP exposes block size, early termination (), query-term pruning (), and many other quantization options.
To keep our theoretical analysis simple and obtain direct theoretical guarantees (to our knowledge, no prior work on learned sparse retrieval jointly establishes theoretical bounds on candidate contraction and Top-K survival), we deliberately avoid adding generic optimizations, such as kNN graph expansion, vector quantization, clustering and so on. We just do not want to add them, rather than our method can not be combined with them.
We sincerely hope the reviewer will reconsider the assessment and update their rating according to our theory and the newly added experiments.
We inspected src/pylib/mod.rs, lines 664–709, in the historical GitHub commit (ba2a...), as well as the official pyseismic-lsr 0.5.2 source distribution. The batch_search() function contains the following code ...
Thanks for inspecting the code. Pls note that batch_search only parallelize across queries (one query/thread). Whether or not the implementation has other issues, your method uses multiple threads per query, so the comparison conflates algorithmic efficiency with parallelization strategy. You configuration also set number of threads > number queries which further amplifies this bias.
The sequence-length setting and FP16 model inference are most possible contributors to the observed differences. Given the time constraints of the rebuttal period and the cost of regenerating embeddings for the full corpus, we have not yet conducted controlled ablations to determine their individual contributions.
Thanks for your explanation. Please make sure you reproduce the same numbers in your next version.
Why F@1000? Due to space limitations, we report F@1000, which measures the fraction of exhaustive SPLADE Top-1000 documents preserved by each method. It directly corresponds to our theoretical ranking-preservation objective and clearly distinguishes the methods.
You can add multiple responses. I also suggest to add P50, P90 latency. Space limitation is not a good reason to not provide critical information.
Instead, we redesigned our index storage and introduced a new method, bitmap-MultiHit, repeating all system comparisons under a single thread (1T/Q1).
This is a new method that wasn't in the original manuscript. The rebuttal period is intended for authors to clarify and support the submitted work. It is not intended for introducing new methods or substantially changing the paper's core contributions, which cannot be adequately reviewed in the discussion period.
Response to Weakness1 about Multithread: New Single-Thread Method
We sincerely thank the reviewer for pointing out the unfairness of comparing our 16T/Q1 implementation against retrieval engines that process each query using a single thread. Motivated by your criticism, we have completely removed multithreaded evaluations and related performance claims from the revised paper. Instead, we redesigned our index storage and introduced a new method, bitmap-MultiHit, repeating all system comparisons under a single thread (1T/Q1).
Bitmap-MultiHit now achieves the lowest single-threaded latency among all evaluated SPLADE retrieval methods, while preserving the original retrieval quality and maintaining substantial storage and memory advantages.
All methods use SPLADE-v3, Top-1000 retrieval, and 1T/Q1. Latency is measured in ms/query; F@1000 is reported as a percentage.
(a) MS MARCO
(b) DBpedia-Entity
(c) FiQA
Why F@1000? Due to space limitations, we report F@1000, which measures the fraction of exhaustive SPLADE Top-1000 documents preserved by each method. It directly corresponds to our theoretical ranking-preservation objective and clearly distinguishes the methods.
Performance improvements. We retain the original Q1–Q3 parameter-selection strategy and operating points: (B,a)=(8,2) on MS MARCO and DBpedia-Entity, and (10,2) on FiQA. These parameters were not retuned for bitmap optimization, and retrieval quality remains unchanged.
Bitmap-MultiHit is 2.21–2.63× faster than the original MultiHit and 2.24–4.92× faster than bitmap-Naive. On MS MARCO and DBpedia-Entity, it outperforms Seismic (
kNN=0) by 1.62–2.23× and BMP by 1.64–3.76×. On FiQA, it is 13.07× faster than PISA MaxScore, the fastest external baseline. TREC19/20 results are included in the revised manuscript.Storage and memory. Bitmap optimization adds only 10–14% storage. Our index remains 2.14–9.17× smaller than Seismic-H and 2.16–2.78× smaller than BMP, while their peak RSS is 1.79–4.25× and 4.24–8.87× ours, respectively.
Implementation and ablation. We cache posting lists as bitmaps, count hits word by word, and represent candidates as masks. Scoring still processes nonempty tiles and maintains the global Top-1000. We also implement bitmap variants of Seismic and BMP: Seismic's visited hash set is replaced with a bitset indexed by vector offsets, and BMP uses a bitmap to skip empty score buckets.
Bitmap optimization accelerates MultiHit by 2.21–2.63× and Naive by 1.15–1.97×, but barely affects Seismic or BMP. This is because bitmap operations directly accelerate MultiHit's hit counting, whereas Seismic relies primarily on clustered traversal, summary pruning, and forward-index scoring, and BMP on score bounds and block processing.
More detailed results have been added to the revised PDF, and the corresponding code is being updated.
Bitmap-MultiHit also preserves the original filtering rule, theoretical motivation, and retrieval quality. We sincerely appreciate your criticism, which led to a substantial improvement in our implementation and evaluation method.
We hope these new results address your concern about the fairness and practicality of our system comparison and encourage you to reconsider your assessment.