- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper targets data agents that answer natural-language questions over heterogeneous enterprise databases. It proposes “latent equivalence learning”: a finite inventory of persistent identities (business roles), whose dataset-specific realizations are learned as Gaussian prototypes from supporting and opposing evidence over evidence objects such as columns, record groups, and relations. A separate query-prototype model and a learned compatibility matrix map each query to activations over these identities. A bounded evidence view is then materialized and passed to an LLM Arbiter and a DuckDB-equipped agent. The theoretical part covers the induced equivalence relation, a membership pseudometric, and a margin–Lipschitz local stability certificate. On DAB (54 queries, 12 datasets), the full system reaches 94.67% stratified Pass@1, versus 84.13% for the same harness without the learned representation and 55.51% for the benchmark’s Opus 4.6 reference agent, ranking first on the leaderboard.
Critical strengths
The result on DAB is strong in absolute terms, and the paper is honest about some limitations, e.g., cost and the conservative verification policy. The intuition of separating reusable structural organization from query-time LLM reasoning is reasonable.
Critical weaknesses
Overall this reads as a strong engineering system wrapped in academic formalism rather than a research contribution that can be evaluated.
First, the parts that actually determine the system are never disclosed: how the identity inventory is defined, how the supporting and opposing evidence weights are constructed (the paper explicitly leaves this unspecified), what the feature map is, what data the prototype and query models are trained on, and the hyperparameters. The code is not released because the system is proprietary. What remains is a generic notation layer that a reader cannot reproduce, verify, or build upon.
Second, the theory adds little. Equivalence classes induced by a function, a pseudometric on softmax profiles, and a standard margin–Lipschitz certificate are all elementary, as the authors themselves acknowledge, and Proposition 3 is essentially a tautology. The method itself is close to semantic column-type detection and schema matching combined with prototypical networks and retrieval routing. Related work on schema linking in text-to-SQL, schema matching and entity resolution, semantic layers, and agent memory is missing, and the markdown memory and skill-file approach the paper argues against is never compared.
Third, the empirical evidence does not support the claims. Everything rests on a single 54-query benchmark with no train/test separation, so it is unclear whether the prototypes and query models were fit on the evaluation datasets. The leaderboard entry is marked as using a DAB-specific tuned prompt and hints, which seems to contradict the reproducibility statement, and there are multiple successive submissions on the same 54 queries. The only ablation (Core vs. full system) changes many things at once: the Arbiter, precomputed query interpretation and resolved-field artifacts, different prompts, and roughly twice the tokens and tool calls with deeper post-answer verification. The 10-point gain therefore cannot be attributed to the learned representation. The 39-point gain over the reference agent largely reflects a stronger model and a different harness. There are no component-level ablations, no compute-matched baseline, no evaluation on other benchmarks or unseen datasets, and no real enterprise data.
Questions for authors
see weakness
Thank you for taking the time to review our manuscript! We appreciate your recognition of the improvement over Core. We agree our original experiments left questions about attribution and cost, and your review made clear where more evidence was needed.
Before addressing your points, we want to add an AI disclosure. We used AI to assist with adapting code and preparing review-specific diagnostics, checking results against our baseline. AI checked, but did not develop, our new theorems. These build on a planned theoretical extension originally omitted to keep our narrative focused. We hope they strengthen the mathematical basis of the architecture.
All references are to our revised manuscript.
We added Flat Retrieval (Sections 4.1–4.2), a familiar RAG-style alternative combining MiniLM cosine similarity and word TF–IDF through reciprocal-rank fusion. Flat indexes the same source-bound records with table, field/relation, and source references, but receives no EQ memberships, plans, candidates, or answers. Both feed the same fresh Sonnet 4.6 Arbiter and Opus 5 agent with the same prompts, tools, source access, settings, and stopping rules. They share an 8,192-token evidence budget and limits of two million total tokens, 150 tool calls, and one final verification.
On the ten-question primary panel, usage was approximately 404k versus 388k tokens and 22.7 versus 21.3 tool calls for EQ and Flat. Yet EQ exposed 70.3% of reference-required evidence versus 31.7%, with higher recall on nine of ten cases. Neither agent saw the evaluator-authored references; recall remains exploratory. Across all 13 cases, we validate 12/13 versus 10/13 (Table 3). All attempts were retained. This examines the evidence interface under comparable resources, not every component. A reconstructed soft-versus-hard test also tied at 2/4. This directly observes your request to see under comparable token budget constraints, can we get reasonable improvement, and the answer is clearly yes. We also did this across multiple runs where the results remaining consistent at 12/13 vs 10/13.
Our construction uses confidence/reliability signals and negative signals to distinguish supporting from opposing evidence. Section 3.4 shows how support builds precision-weighted prototypes and attraction losses; opposition penalizes proximity to incompatible identities. We cannot disclose our proprietary weights, but the mathematical uses of these signals are now explicit. Appendix C.4 derives a conditional loss-to-margin result connecting them to assignment separation.
We also agree that elementary equivalence relations or stability certificates alone do not justify the architecture. Theorem 1 characterizes the coarsest decision-preserving partition and bounds regret from merging distinguishable contexts. Theorem 2 identifies when hardening soft role profiles discards decision-relevant information. Full proofs appear in Appendix D. These are conditional guarantees, not benchmark accuracy claims.
This is the sense in which query-side structure transfers: a request for a customer identifier activates the same recurring role even when different organizations use different schemas. Query requirements can be reused because their identity coordinates remain shared, while evidence-side realizations are learned locally. We did not design a frozen-model transfer test: our intended setting is learning from an organization’s data before querying it. We agree that unseen-database transfer without adaptation requires a separate experiment, and our conclusion does not claim it. Instead, our system was built around the concept that enterprise data that is incoming will remain consistent in type and structure, these signals do transfer in our system.
Thank you again for your suggestions. They prompted a stronger comparison and clarified the scope of our theory and experiments.