acceptodds
Withdrawn from ICLR 2027

The Conjunctive Fraction of Embodied Benchmarks: Composition Cannot Be Measured Where the Write Path Has Already Destroyed It

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The Capability Convergence Hypothesis (CCH) holds that capability is purchased by access structure: the joint possession of a compressive bounded-state channel and a scalable verbatim-index channel. It proves composition strictly super-additive on a witness family that obstructs each channel separately, and names as its largest open risk a measurement it reports "has not been run by anyone, including us": how often natural workloads instantiate that joint load. We run it for embodied multimodal streams. An arithmetic criterion, computable before any run, first decides whether a benchmark can exhibit the effect at all: a bounded score cannot host an interaction whose components already sum past the bound. A census of public video question-answering corpora then puts the conjunctive fraction in the low single digits, so solving that subset perfectly would move the richest benchmark by less than the run-to-run floor at which these benchmarks are compared. We supply the witness the hypothesis implies (735 judge-free items over head-mounted kitchen streams), and in the registered grid every prediction fails and the channel dichotomy reverses. A diagnostic locates the failure upstream of every arm, in the shared single-frame write path, which decides the locator verb only 11-23 the ledger from annotations instead, composition by address beats a same-cost different-occurrence control by 14-26$ points on every backbone and is null on both single-axis families (the corpus is a valid witness and the loss is on the write), yet even then it answers less than half.

Primary area:
datasets and benchmarks
Keywords:
embodied multimodal benchmarks, benchmark diagnosis, compositional question answering, memory architecture, access structure, super-additivity, egocentric video, pre-registration, annotation-only instrument, EPIC-KITCHENS

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NRqdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discuss the topic of "Composition Cannot Be Measured Where the Write Path Has Already Destroyed It".

Critical strengths

The paper at least has enough pages to discuss the topic. The problem seems important.

Critical weaknesses

I am not an expert in this area, so these comments reflect my preliminary assessment. I will follow the discussion with other reviewers and remain open to updating my evaluation.

  1. The paper devotes substantial space to discussing its hypothesis, but I remain unclear about its implications for embodied benchmarks. What problems does it actually solve? The writing is also unconventional. For example, “has not been run by anyone, including us” in the abstract provides little useful information.

  2. The writing style makes the paper difficult to follow. Most sentences in the abstract span three to four lines, and the excessive use of parentheses gives the text an AI-generated impression.

  3. Figure 1 is confusing rather than informative. In particular, I do not understand why “what follows” occupies so much space when it is not a central contribution.

  4. Apart from Figure 1, the paper provides almost no pipeline diagrams or conceptual illustrations to help readers understand its ideas.

I spent several hours reading this paper but still struggled to understand its main contributions. While this may partly reflect my limited expertise, I believe the paper needs substantial improvements in clarity and presentation.

Questions for authors

None

Reviewer_JXPhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study whether current benchmarks can measure the benefit of combining a compact memory with retrieval. They estimate how many questions require both, derive a bound on retrieval alone, and construct such questions from videos. They report that using an event ledger built from annotations improves accuracy over a control that retrieves a different occurrence of the same event.

Critical strengths

The video experiment is a useful diagnostic. In Table 8, the authors report that using the annotation-derived ledger for address-based retrieval significantly improves accuracy points over retrieving a different occurrence of the same event at equal cost. This suggests that the implemented pipeline loses information needed to answer the questions. I would have liked to reproduce this result - but some necessary records are not included in the submission.

To anchor the review, I reran the authors' scoring code was on their saved predictions and experimental configurations, including the original and revised question sets to verify correctness - and the results and data matches the claimed numbers.

Critical weaknesses
  1. The main prevalence claim converts a lower bound into an upper bound. The authors' annotation-only “rule pass” in Section 3 and Table 2 labels 233 of 45,728 questions as conjunctive. Summing their released counts gives 32,271 unclassified questions by the authors metrics. The underlying question files are absent, so I could not rerun the classification itself. Even assuming every classified label is correct, those counts permit a conjunctive fraction anywhere from 233/45,728=0.51%233/45,728=0.51\% to (233+32,271)/45,728=71.08%(233+32,271)/45,728=71.08\%. Calling 0.51% “at most” is therefore simply incorrect given the data the authors present. Appendix G repeats this reversal. The reported counts do not establish the central claim that existing benchmarks contain too few conjunctive questions to measure composition.

  2. The retrieval bound does not certify the witness, and the positive result cannot be recomputed. Assumption 1 limits the chance that the requested occurrence ranks first. The proof on pages 40–41 assumes that the entire retrieved set, including timestamps, leaves the answer indistinguishable. These are different conditions. This can be tested with a counterexample: take six independently ranked occurrences with distinct successor values and retrieve four occurrence/successor pairs using eight frames. Enumerating all 720 rankings gives probability 1/61/6 that the target ranks first, but probability (42)/(62)=0.4\binom{4}{2}/\binom{6}{2}=0.4 that the second-earliest retrieved occurrence is the true second occurrence. Timestamps convey information even when the top rank does not. Definition 1(i)'s stronger restriction may exclude this example, but the paper does not clarify this restriction for its actual task. The retrieval bound therefore does not certify the proposed witness.

    The records needed to check the positive result are missing. Appendix O, p. 44, states: “Those records are released for every cell.” The submitted supplement contains neither the oracle per-question records and address traces nor the named AXIS runner and decomposition script. The authors' reported accuracy improvement therefore could not be recomputed from the supplied material.

Questions for authors
  1. What bounds the number of conjunctive questions in the unclassified set, and what establishes that the full retrieval output leaves the answer indistinguishable? Without these, the paper can not establish its claims about benchmark blindness or the proposed witness.

  2. Compositional video questions and replacing predicted intermediate representations with ground truth are established approaches. Please consider these two previous works and explain what the bounded-memory setting actually adds:

  3. The references contain hallucinated author details. Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos lists Shangzhe Xu and Sifan Yang; the actual authors are Qirui Chen, Shangzhe Di and Weidi Xie. LongVidSearch is attributed to Ruoyu Yang and “Others”; its authors are Rongyi Yu, Chenyuan Duan and Wentao Zhang. There may be others.

Reviewer_1ARQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper builds on the "Capability Convergence Hypothesis", which means capability comes from combining a compressive bounded-state channel and a verbatim-index channel. In this case, locator is a reduction over the stream and value is a single-moment percept. The paper contributes (1) an "availability" observation: additive super-additivity Cap(Cs+Cv) > Cap(Cs)+Cap(Cv)−Cap(∅) is impossible when the right-hand side ≥ 1; (2) a rule-based, annotation-only census of six video QA corpora (45,728 questions), finding 0.51% conjunctive questions under a rule pass that leaves 71% unclassified; (3) a generated witness corpus of 735 (later 308×3) multiple-choice items from 8 EPIC-KITCHENS-100 streams, with families RET, TRK, CONJ; (4) a pre-registered grid of four context-assembly arms on three open backbones (12–31B), in which every registered prediction fails and all CONJ arms sit near chance; (5) a diagnostic attributing the failure to a single-frame "write path" (verb accuracy 11–23%), and an oracle-ledger experiment where dereferencing the annotated k-th occurrence beats dereferencing a wrong occurrence by 14–26 points.

Critical strengths
  1. Unusually transparent reporting. Predictions were frozen before runs, every failed criterion is reported, defects in the authors' own generator (raw-string vs. class occurrence semantics) are disclosed and re-run, and per-question records and analyzers are promised. This is a commendable practice.
  2. A potentially useful benchmark-design observation. Questions whose locator is an ordinal over the stream ("the k-th time you open the fridge") are rare in existing video QA corpora, and such items can be generated with judge-free gold from dense action annotations. This could be useful for memory-architecture evaluation if developed into a properly validated benchmark.
Critical weaknesses
  1. The central quantitative claim inverts the direction of its own bound. The census counts every unclassified query (71%) as non-conjunctive, so 0.51% is, at best, a lower bound on the conjunctive fraction (and only if the rule's precision is high, which is not measured: no human validation or inter-annotator agreement). The supplementary script itself (analysis/a0_conjfrac_probe.py) is labeled "FIRST-CUT PROBE (not the registered instrument): lower-bound the conjunctive fraction". It consists of three keyword regexes, and its locator regex fires on words such as "before", "after", "next", "other", "again" and "most". That contradicts the paper's own classification of NExT-QA "before/after" questions as index-shaped (App. H), so even the precision behind the 233 positives is doubtful. The input pages it reads (a2_pages/) are not included, so the census cannot be re-run from the supplement. Yet the paper writes "p_C ≤ 0.51%" throughout and uses it in §3.1 to conclude that "these benchmarks are blind, on a measured lower bound, at the effect size access structure can produce". The detectability argument requires an upper bound on p_C, which the instrument cannot provide. The run-to-run floor f = 9.0 points that sets the 10.3% break-even is imported from the authors' own concurrent submission (per notes/A0_DESIGN.md, measured on that submission's synthetic SEW corpus) and reflects their specific (weak) pipeline, not the noise level at which the field compares memory systems. The "the field's benchmarks are blind" conclusion is therefore unsupported. The remaining theoretical content (Prop. 1, "a bounded score cannot exceed 1") is, as the authors acknowledge, immediate.
  2. The empirical study does not test what it sets out to test, and its positive result is close to tautological. All 36 grid cells on CONJ are within a few points of chance (12.5%), the shared single-frame writer recognizes the action verb only 11–23% of the time, the RET family leaks to a question-only baseline, the TRK blind arm falls below chance, and one of three backbones is unscorable on many cells. The registered v1/v2 generator was wrong on one third to one half of items. With such a write path, the grid says nothing about composition of memory channels. The "witness validity" result shows that, given the corpus's own annotation ledger, dereferencing the annotated correct frame beats dereferencing a different occurrence of the same event. That establishes only that the answer is visible in the correct frame (which holds by construction of the items), not that the composite channels are super-additive. The authors themselves decline to call it super-additivity. The study uses one domain, 8 streams, one budget, and frozen ≤31B models with no frontier model or standard baseline (e.g., uniform frame sampling, which the authors note is missing).
Questions for authors
  1. Given that unclassified items are counted as non-conjunctive, and that your own script describes itself as a lower bound, on what basis is 0.51% an upper bound? Can you provide a human-annotated random sample (with agreement) estimating the true conjunctive fraction among unclassified items, and the precision of the regex rule on the 233 positives (given that it matches "before/after" locators you classify as index-shaped)?
  2. What run-to-run variance do standard long-video memory systems (e.g., published agents on Video-MME/LVBench) exhibit? Is 9.0 points representative beyond your companion pipeline?
  3. Does any deployable (non-oracle) arm exceed BLIND on CONJ by a statistically significant margin on any backbone?
  4. How does this paper differ in content and contribution from the five concurrent anonymous submissions (2026a–e), in particular 2026d (which already proves the conjunctive proposition and runs the grid) and 2026e (which measures the write path on these same cached streams)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.